[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:03 dist-test-slave-srgv krb5kdc[29012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:03 dist-test-slave-srgv krb5kdc[29012](info): set up 2 sockets
Aug 10 18:01:03 dist-test-slave-srgv krb5kdc[29012](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 18:01:04.945533 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:05 dist-test-slave-srgv krb5kdc[29012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:05.077502 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:42279
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:05.197866 29026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:05.198325 29026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:05.198467 29026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:05.202685 29026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:05.202798 29026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:05.202838 29026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:05.202874 29026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:05.202919 29026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:05.209225 29026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:42279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:05.210898 29026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:05.212149 29026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:05.223168 29031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:05.223166 29032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:05.223663 29034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:05.225200 29026 server_base.cc:1034] running on GCE node
I20260810 18:01:05.226006 29026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:05.227738 29026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:05.228936 29026 hybrid_clock.cc:648] HybridClock initialized: now 1786384865228887 us; error 73 us; skew 500 ppm
Aug 10 18:01:05 dist-test-slave-srgv krb5kdc[29012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384865, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:05.232077 29026 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:05.233320 29026 webserver.cc:460] Webserver started at http://127.28.83.126:40339/ using document root <none> and password file <none>
I20260810 18:01:05.234246 29026 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:05.234355 29026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:05.234787 29026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:05.237723 29026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "b905f6ea5e1744f48da884d5543f4231"
format_stamp: "Formatted at 2026-08-10 18:01:05 on dist-test-slave-srgv"
server_key: "04a237f3369ac1e09efb95121f538abe"
server_key_iv: "689d15f1b98b5a3fb54b949817e7519c"
server_key_version: "encryptionkey@0"
I20260810 18:01:05.238647 29026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "b905f6ea5e1744f48da884d5543f4231"
format_stamp: "Formatted at 2026-08-10 18:01:05 on dist-test-slave-srgv"
server_key: "04a237f3369ac1e09efb95121f538abe"
server_key_iv: "689d15f1b98b5a3fb54b949817e7519c"
server_key_version: "encryptionkey@0"
I20260810 18:01:05.243865 29026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 18:01:05.246994 29041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:05.248579 29026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:01:05.248819 29026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "b905f6ea5e1744f48da884d5543f4231"
format_stamp: "Formatted at 2026-08-10 18:01:05 on dist-test-slave-srgv"
server_key: "04a237f3369ac1e09efb95121f538abe"
server_key_iv: "689d15f1b98b5a3fb54b949817e7519c"
server_key_version: "encryptionkey@0"
I20260810 18:01:05.249001 29026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:05.264293 29026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:05.267935 29026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:05.268229 29026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:05.278569 29026 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:42279
I20260810 18:01:05.278613 29093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:42279 every 8 connection(s)
I20260810 18:01:05.279908 29026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:05.284873 29094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:05.285076 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 29026
I20260810 18:01:05.285265 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:05.286700 29005 external_mini_cluster.cc:1428] Setting key 2e881dd91cb0ebcab4d1bf383579a094
I20260810 18:01:05.294015 29094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231: Bootstrap starting.
I20260810 18:01:05.298462 29094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:05.299620 29094 log.cc:824] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:05.302770 29094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231: No bootstrap required, opened a new log
I20260810 18:01:05.308779 29094 raft_consensus.cc:348] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b905f6ea5e1744f48da884d5543f4231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42279 } }
I20260810 18:01:05.309257 29094 raft_consensus.cc:374] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:05.309362 29094 raft_consensus.cc:729] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b905f6ea5e1744f48da884d5543f4231, State: Initialized, Role: FOLLOWER
I20260810 18:01:05.310357 29094 consensus_queue.cc:260] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b905f6ea5e1744f48da884d5543f4231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42279 } }
I20260810 18:01:05.310637 29094 raft_consensus.cc:388] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:05.310786 29094 raft_consensus.cc:482] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:05.310927 29094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 18:01:05 dist-test-slave-srgv krb5kdc[29012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:05.311978 29094 raft_consensus.cc:504] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b905f6ea5e1744f48da884d5543f4231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42279 } }
I20260810 18:01:05.312522 29094 leader_election.cc:302] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b905f6ea5e1744f48da884d5543f4231; no voters:
I20260810 18:01:05.312951 29094 leader_election.cc:288] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:05.313064 29099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:05.313268 29099 raft_consensus.cc:686] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [term 1 LEADER]: Becoming Leader. State: Replica: b905f6ea5e1744f48da884d5543f4231, State: Running, Role: LEADER
I20260810 18:01:05.313705 29099 consensus_queue.cc:237] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b905f6ea5e1744f48da884d5543f4231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42279 } }
I20260810 18:01:05.314822 29094 sys_catalog.cc:564] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:05.316689 29100 sys_catalog.cc:455] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b905f6ea5e1744f48da884d5543f4231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b905f6ea5e1744f48da884d5543f4231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42279 } } }
I20260810 18:01:05.316689 29101 sys_catalog.cc:455] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b905f6ea5e1744f48da884d5543f4231. Latest consensus state: current_term: 1 leader_uuid: "b905f6ea5e1744f48da884d5543f4231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b905f6ea5e1744f48da884d5543f4231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42279 } } }
I20260810 18:01:05.316843 29100 sys_catalog.cc:458] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:05.316869 29101 sys_catalog.cc:458] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:05.319878 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:05.293340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.28.83.126:42279)
0810 18:01:05.293836 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:05.293850 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:05.293922 (+ 72us) server_negotiation.cc:407] Connection header received
0810 18:01:05.298216 (+ 4294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:05.298232 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:05.298701 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:05.298985 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:05.302350 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.303638 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.305663 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.306235 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.306385 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:05.312149 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:05.312164 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:05.312181 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:05.312223 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:05.314555 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:05.315255 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:05.315261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:05.315266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:05.315427 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:05.315998 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:05.316002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:05.316015 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:05.316835 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:05.317084 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:05.318305 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0810 18:01:05.318698 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":142,"threads_started":1}
W20260810 18:01:05.321697 29115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:01:05.321827 29115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:01:05.323024 29113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:05.324294 29113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:05.331341 29113 catalog_manager.cc:1347] Generated new cluster ID: 16ce97ed1b8a4a7cb682a8e63f2650d9
I20260810 18:01:05.331434 29113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:05.386314 29113 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:05.387364 29113 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:05.403483 29113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b905f6ea5e1744f48da884d5543f4231: Generated new TSK 0
I20260810 18:01:05.404464 29113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:05.440966 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42279
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:05.595456 29121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:05.595788 29121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:05.595839 29121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:05.600281 29121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:05.600360 29121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:05.600455 29121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:05.606484 29121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42279
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:05.607729 29121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:05.608901 29121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:05.620437 29128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:05.620563 29121 server_base.cc:1034] running on GCE node
W20260810 18:01:05.620362 29127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:05.620422 29131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:05.621130 29121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:05.621841 29121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:05.623021 29121 hybrid_clock.cc:648] HybridClock initialized: now 1786384865622988 us; error 47 us; skew 500 ppm
Aug 10 18:01:05 dist-test-slave-srgv krb5kdc[29012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384865, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:05.627189 29121 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:05.628150 29121 webserver.cc:460] Webserver started at http://127.28.83.65:33003/ using document root <none> and password file <none>
I20260810 18:01:05.628724 29121 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:05.628818 29121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:05.629057 29121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:05.631373 29121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "5870e42276334e4c80484a8e18cfece7"
format_stamp: "Formatted at 2026-08-10 18:01:05 on dist-test-slave-srgv"
server_key: "dc1f129fef8745b63da264ba0ccaf93c"
server_key_iv: "846791ef25df1b8dbd957436cfcbd4f6"
server_key_version: "encryptionkey@0"
I20260810 18:01:05.631999 29121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "5870e42276334e4c80484a8e18cfece7"
format_stamp: "Formatted at 2026-08-10 18:01:05 on dist-test-slave-srgv"
server_key: "dc1f129fef8745b63da264ba0ccaf93c"
server_key_iv: "846791ef25df1b8dbd957436cfcbd4f6"
server_key_version: "encryptionkey@0"
I20260810 18:01:05.636161 29121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 18:01:05.638962 29138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:05.640285 29121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:01:05.640446 29121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "5870e42276334e4c80484a8e18cfece7"
format_stamp: "Formatted at 2026-08-10 18:01:05 on dist-test-slave-srgv"
server_key: "dc1f129fef8745b63da264ba0ccaf93c"
server_key_iv: "846791ef25df1b8dbd957436cfcbd4f6"
server_key_version: "encryptionkey@0"
I20260810 18:01:05.640584 29121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:05.656045 29121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:05.660048 29121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:05.660333 29121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:05.661142 29121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:05.662364 29121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:05.662437 29121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:05.662514 29121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:05.662559 29121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:05.681835 29121 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:41863
I20260810 18:01:05.681862 29251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:41863 every 8 connection(s)
I20260810 18:01:05.682891 29121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:05.689808 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 29121
I20260810 18:01:05.689946 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:05.690227 29005 external_mini_cluster.cc:1428] Setting key f63538b5c5ad6f9c17884e9026e0d316
Aug 10 18:01:05 dist-test-slave-srgv krb5kdc[29012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384865, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:05.700181 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:05.685318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:48095 (local address 127.28.83.126:42279)
0810 18:01:05.685521 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:05.685526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:05.686678 (+ 1152us) server_negotiation.cc:407] Connection header received
0810 18:01:05.688414 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:05.688416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:05.688468 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:05.688550 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:05.689940 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.690730 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.692139 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.692558 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.692690 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:05.695766 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:05.695772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:05.695777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:05.695812 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:05.697573 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:05.698338 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:05.698342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:05.698344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:05.698458 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:05.698911 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:05.698913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:05.698914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:05.699103 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:05.699180 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:05.699933 (+ 753us) server_negotiation.cc:299] Negotiation successful
0810 18:01:05.700047 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 18:01:05.701858 29254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:05.685789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42279 (local address 127.28.83.65:48095)
0810 18:01:05.686492 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:05.686539 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:05.688185 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:05.688729 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:05.688737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:05.689185 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:05.689770 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:05.689784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.690883 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.690887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:05.692006 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:05.692017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.692721 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.692724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:05.692932 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:05.693778 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:05.693795 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:05.695545 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:05.697702 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:05.697712 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:05.697733 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:05.698170 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:05.698563 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:05.698569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:05.698574 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:05.698804 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:05.699218 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:05.699229 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:05.699628 (+ 399us) client_negotiation.cc:769] Sending connection context
0810 18:01:05.699885 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 18:01:05.700308 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":259,"threads_started":1}
I20260810 18:01:05.703306 29252 heartbeater.cc:347] Connected to a master server at 127.28.83.126:42279
I20260810 18:01:05.703646 29252 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:01:05.704373 29252 heartbeater.cc:511] Master 127.28.83.126:42279 requested a full tablet report, sending...
I20260810 18:01:05.706346 29058 ts_manager.cc:194] Registered new tserver with Master: 5870e42276334e4c80484a8e18cfece7 (127.28.83.65:41863)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:05.707957 29058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:48095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:05.728824 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42279
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:05.871150 29257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:05.871450 29257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:05.871531 29257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:05.875861 29257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:05.875932 29257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:05.876030 29257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:05.881906 29257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42279
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:05.883065 29257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:05.884202 29257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:01:05.896106 29257 server_base.cc:1034] running on GCE node
W20260810 18:01:05.896068 29264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:05.896131 29262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:05.896415 29266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:05.896843 29257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:05.897616 29257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:05.898782 29257 hybrid_clock.cc:648] HybridClock initialized: now 1786384865898763 us; error 30 us; skew 500 ppm
Aug 10 18:01:05 dist-test-slave-srgv krb5kdc[29012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384865, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:05.902213 29257 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:05.903152 29257 webserver.cc:460] Webserver started at http://127.28.83.66:37937/ using document root <none> and password file <none>
I20260810 18:01:05.903724 29257 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:05.903787 29257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:05.904045 29257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:05.906404 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "0c93293a767f49d5b6522545ceb6e4e6"
format_stamp: "Formatted at 2026-08-10 18:01:05 on dist-test-slave-srgv"
server_key: "b15fbe0cb9c06afb2ba82c4a5829cc82"
server_key_iv: "0bb35347ac54884dcf94c42f28826783"
server_key_version: "encryptionkey@0"
I20260810 18:01:05.907051 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "0c93293a767f49d5b6522545ceb6e4e6"
format_stamp: "Formatted at 2026-08-10 18:01:05 on dist-test-slave-srgv"
server_key: "b15fbe0cb9c06afb2ba82c4a5829cc82"
server_key_iv: "0bb35347ac54884dcf94c42f28826783"
server_key_version: "encryptionkey@0"
I20260810 18:01:05.911031 29257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:01:05.913592 29275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:05.914836 29257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:01:05.914970 29257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "0c93293a767f49d5b6522545ceb6e4e6"
format_stamp: "Formatted at 2026-08-10 18:01:05 on dist-test-slave-srgv"
server_key: "b15fbe0cb9c06afb2ba82c4a5829cc82"
server_key_iv: "0bb35347ac54884dcf94c42f28826783"
server_key_version: "encryptionkey@0"
I20260810 18:01:05.915086 29257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:05.925740 29257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:05.928895 29257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:05.929174 29257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:05.929984 29257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:05.930974 29257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:05.931022 29257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:05.931172 29257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:05.931242 29257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:05.952876 29257 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:39687
I20260810 18:01:05.954690 29390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:39687 every 8 connection(s)
I20260810 18:01:05.955937 29257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:05.957226 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 29257
I20260810 18:01:05.957340 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:05.957662 29005 external_mini_cluster.cc:1428] Setting key 9b75942693ea40d1018206607203e6a8
Aug 10 18:01:05 dist-test-slave-srgv krb5kdc[29012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384865, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
I20260810 18:01:05.975210 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:05.958747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:55315 (local address 127.28.83.126:42279)
0810 18:01:05.958926 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:05.958931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:05.960644 (+ 1713us) server_negotiation.cc:407] Connection header received
0810 18:01:05.962091 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:05.962094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:05.962179 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:05.962291 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:05.964937 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.965844 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.967512 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.967740 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.967905 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:05.971186 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:05.971190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:05.971192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:05.971216 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:05.972938 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:05.973615 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:05.973619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:05.973621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:05.973709 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:05.974078 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:05.974081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:05.974083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:05.974276 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:05.974387 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:05.974911 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 18:01:05.975041 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:01:05.976150 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:05.959223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42279 (local address 127.28.83.66:55315)
0810 18:01:05.960465 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:05.960514 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:05.961846 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:05.962513 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:05.962533 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:05.963495 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:05.964625 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:05.964660 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.966014 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.966017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:05.967384 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:05.967394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:05.967956 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:05.967960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:05.968210 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:05.968952 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:05.968969 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:05.970980 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:05.973142 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:05.973147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:05.973160 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:05.973504 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:05.973807 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:05.973810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:05.973813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:05.973960 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:05.974385 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:05.974393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:05.974654 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 18:01:05.974848 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 18:01:05.975210 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":797,"threads_started":1}
I20260810 18:01:05.977598 29391 heartbeater.cc:347] Connected to a master server at 127.28.83.126:42279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:01:05.977953 29391 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:05.978456 29391 heartbeater.cc:511] Master 127.28.83.126:42279 requested a full tablet report, sending...
I20260810 18:01:05.979637 29058 ts_manager.cc:194] Registered new tserver with Master: 0c93293a767f49d5b6522545ceb6e4e6 (127.28.83.66:39687)
I20260810 18:01:05.980228 29058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:55315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:01:05.996671 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42279
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:06.163668 29396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:06.163996 29396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:06.164062 29396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:06.168556 29396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:06.168648 29396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:06.168712 29396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:06.175151 29396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42279
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:06.176412 29396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:06.177524 29396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:06.189394 29402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:06.189360 29404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:06.189459 29401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:06.189884 29396 server_base.cc:1034] running on GCE node
I20260810 18:01:06.190339 29396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:06.191033 29396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:06.192253 29396 hybrid_clock.cc:648] HybridClock initialized: now 1786384866192218 us; error 47 us; skew 500 ppm
Aug 10 18:01:06 dist-test-slave-srgv krb5kdc[29012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384866, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:06.196027 29396 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:06.197237 29396 webserver.cc:460] Webserver started at http://127.28.83.67:45753/ using document root <none> and password file <none>
I20260810 18:01:06.197924 29396 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:06.198026 29396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:06.198297 29396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:06.200755 29396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "5ded6c2d80344947b0d6e956dbfd291b"
format_stamp: "Formatted at 2026-08-10 18:01:06 on dist-test-slave-srgv"
server_key: "58747f55fd8c1d6b3b6db9e9929ba808"
server_key_iv: "dad37512106c914a541317ffb00d9d9a"
server_key_version: "encryptionkey@0"
I20260810 18:01:06.201480 29396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "5ded6c2d80344947b0d6e956dbfd291b"
format_stamp: "Formatted at 2026-08-10 18:01:06 on dist-test-slave-srgv"
server_key: "58747f55fd8c1d6b3b6db9e9929ba808"
server_key_iv: "dad37512106c914a541317ffb00d9d9a"
server_key_version: "encryptionkey@0"
I20260810 18:01:06.205783 29396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:01:06.208587 29411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:06.209995 29396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:06.210147 29396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "5ded6c2d80344947b0d6e956dbfd291b"
format_stamp: "Formatted at 2026-08-10 18:01:06 on dist-test-slave-srgv"
server_key: "58747f55fd8c1d6b3b6db9e9929ba808"
server_key_iv: "dad37512106c914a541317ffb00d9d9a"
server_key_version: "encryptionkey@0"
I20260810 18:01:06.210276 29396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:06.224449 29396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:06.229162 29396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:06.229532 29396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:06.230387 29396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:06.231374 29396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:06.231449 29396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:06.231516 29396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:06.231546 29396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:06.255280 29396 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:34755
I20260810 18:01:06.255352 29530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:34755 every 8 connection(s)
I20260810 18:01:06.256364 29396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:06.257068 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 29396
I20260810 18:01:06.257232 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:06.257575 29005 external_mini_cluster.cc:1428] Setting key 725e557fd7a63741114793c3b8b18222
Aug 10 18:01:06 dist-test-slave-srgv krb5kdc[29012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384866, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:06.272818 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:06.258686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:59065 (local address 127.28.83.126:42279)
0810 18:01:06.258830 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:06.258834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:06.260268 (+ 1434us) server_negotiation.cc:407] Connection header received
0810 18:01:06.261918 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:06.261920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:06.261967 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:06.262036 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:06.263434 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:06.263966 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:06.265298 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:06.265661 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:06.265772 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:06.268669 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:06.268672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:06.268675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:06.268702 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:06.270431 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:06.270953 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:06.270956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:06.270958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:06.271057 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:06.271544 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:06.271549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:06.271554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:06.271821 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:06.271940 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:06.272536 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 18:01:06.272688 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:01:06.273849 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:06.259216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42279 (local address 127.28.83.67:59065)
0810 18:01:06.259978 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:06.260041 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:06.261719 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:06.262178 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:06.262185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:06.262678 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:06.263296 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:06.263313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:06.264095 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:06.264097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:06.265180 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:06.265191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:06.265807 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:06.265810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:06.266015 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:06.266653 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:06.266696 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:06.268476 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:06.270541 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:06.270546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:06.270561 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:06.270847 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:06.271318 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:06.271320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:06.271322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:06.271474 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:06.271951 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:06.271959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:06.272267 (+ 308us) client_negotiation.cc:769] Sending connection context
0810 18:01:06.272480 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 18:01:06.272804 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":208,"threads_started":1}
I20260810 18:01:06.275178 29531 heartbeater.cc:347] Connected to a master server at 127.28.83.126:42279
I20260810 18:01:06.275434 29531 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:06.275977 29531 heartbeater.cc:511] Master 127.28.83.126:42279 requested a full tablet report, sending...
I20260810 18:01:06.277119 29058 ts_manager.cc:194] Registered new tserver with Master: 5ded6c2d80344947b0d6e956dbfd291b (127.28.83.67:34755)
I20260810 18:01:06.277793 29058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:59065
I20260810 18:01:06.284700 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:01:06 dist-test-slave-srgv krb5kdc[29012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:01:06 dist-test-slave-srgv krb5kdc[29012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:06.314647 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:06.304128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.28.83.65:41863)
0810 18:01:06.304415 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:06.304428 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:06.304518 (+ 90us) server_negotiation.cc:407] Connection header received
0810 18:01:06.304619 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:06.304624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:06.304811 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:06.304937 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:06.305935 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:06.306785 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:06.307691 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:06.308110 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:06.308199 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:06.310896 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:06.310906 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:06.310919 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:06.310944 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:06.312735 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:06.313181 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:06.313187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:06.313188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:06.313269 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:06.313625 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:06.313628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:06.313629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:06.313870 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:06.314029 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:06.314283 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 18:01:06.314411 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
W20260810 18:01:06.315747 29166 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48666
Aug 10 18:01:06 dist-test-slave-srgv krb5kdc[29012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:01:06 dist-test-slave-srgv krb5kdc[29012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:06.343834 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:06.335286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.28.83.65:41863)
0810 18:01:06.335454 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:06.335460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:06.335586 (+ 126us) server_negotiation.cc:407] Connection header received
0810 18:01:06.335783 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:06.335787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:06.335831 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:06.335924 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:06.336680 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:06.337206 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:06.338138 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:06.338373 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:06.338492 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:06.341061 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:06.341064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:06.341066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:06.341092 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:06.342435 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:06.342836 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:06.342839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:06.342840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:06.342912 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:06.343154 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:06.343156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:06.343157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:06.343318 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:06.343419 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:06.343609 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 18:01:06.343711 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:01:06.345006 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 29121
I20260810 18:01:06.351934 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 29257
I20260810 18:01:06.359896 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 29396
I20260810 18:01:06.366747 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 29026
2026-08-10T18:01:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2518 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:06 dist-test-slave-srgv krb5kdc[29545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:06 dist-test-slave-srgv krb5kdc[29545](info): set up 2 sockets
Aug 10 18:01:06 dist-test-slave-srgv krb5kdc[29545](info): commencing operation
krb5kdc: starting...
W20260810 18:01:07.431978 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:07 dist-test-slave-srgv krb5kdc[29545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:07.561223 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:39851
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:39851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:07.674126 29558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:07.674484 29558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:07.674559 29558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:07.678728 29558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:07.678838 29558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:07.678880 29558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:07.678933 29558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:07.678972 29558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:07.684823 29558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:39851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:39851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:07.686079 29558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:07.687178 29558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:07.695870 29567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:07.695887 29564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:07.695942 29565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:07.696095 29558 server_base.cc:1034] running on GCE node
I20260810 18:01:07.696722 29558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:07.697913 29558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:07.699103 29558 hybrid_clock.cc:648] HybridClock initialized: now 1786384867699085 us; error 31 us; skew 500 ppm
Aug 10 18:01:07 dist-test-slave-srgv krb5kdc[29545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384867, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:07.709308 29558 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:07.710489 29558 webserver.cc:460] Webserver started at http://127.28.83.126:42639/ using document root <none> and password file <none>
I20260810 18:01:07.711103 29558 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:07.711174 29558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:07.711352 29558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:07.713455 29558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "e01cb4e0adb94f91bbbcb0e566c1b124"
format_stamp: "Formatted at 2026-08-10 18:01:07 on dist-test-slave-srgv"
server_key: "3294237743641ceb751038188d912c53"
server_key_iv: "6a215a56f80b987b904f71b722dc7787"
server_key_version: "encryptionkey@0"
I20260810 18:01:07.714007 29558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "e01cb4e0adb94f91bbbcb0e566c1b124"
format_stamp: "Formatted at 2026-08-10 18:01:07 on dist-test-slave-srgv"
server_key: "3294237743641ceb751038188d912c53"
server_key_iv: "6a215a56f80b987b904f71b722dc7787"
server_key_version: "encryptionkey@0"
I20260810 18:01:07.717738 29558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:01:07.720155 29574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:07.721527 29558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:07.721652 29558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "e01cb4e0adb94f91bbbcb0e566c1b124"
format_stamp: "Formatted at 2026-08-10 18:01:07 on dist-test-slave-srgv"
server_key: "3294237743641ceb751038188d912c53"
server_key_iv: "6a215a56f80b987b904f71b722dc7787"
server_key_version: "encryptionkey@0"
I20260810 18:01:07.721767 29558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:07.737134 29558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:07.740237 29558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:07.740463 29558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:07.748112 29558 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:39851
I20260810 18:01:07.748142 29628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:39851 every 8 connection(s)
I20260810 18:01:07.749179 29558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:07.752111 29629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:07.757944 29629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124: Bootstrap starting.
I20260810 18:01:07.758343 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 29558
I20260810 18:01:07.758440 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:07.758637 29005 external_mini_cluster.cc:1428] Setting key 18be095d694e36c15f3a1232a7bb0679
I20260810 18:01:07.760706 29629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:07.761534 29629 log.cc:824] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:07.763589 29629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124: No bootstrap required, opened a new log
I20260810 18:01:07.767359 29629 raft_consensus.cc:348] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01cb4e0adb94f91bbbcb0e566c1b124" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39851 } }
I20260810 18:01:07.767632 29629 raft_consensus.cc:374] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:07.767731 29629 raft_consensus.cc:729] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01cb4e0adb94f91bbbcb0e566c1b124, State: Initialized, Role: FOLLOWER
Aug 10 18:01:07 dist-test-slave-srgv krb5kdc[29545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:07.768301 29629 consensus_queue.cc:260] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01cb4e0adb94f91bbbcb0e566c1b124" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39851 } }
I20260810 18:01:07.768489 29629 raft_consensus.cc:388] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:07.768572 29629 raft_consensus.cc:482] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:07.768685 29629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:07.770045 29629 raft_consensus.cc:504] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01cb4e0adb94f91bbbcb0e566c1b124" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39851 } }
I20260810 18:01:07.770424 29629 leader_election.cc:302] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e01cb4e0adb94f91bbbcb0e566c1b124; no voters:
I20260810 18:01:07.770798 29629 leader_election.cc:288] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:07.770927 29634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:07.771170 29634 raft_consensus.cc:686] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [term 1 LEADER]: Becoming Leader. State: Replica: e01cb4e0adb94f91bbbcb0e566c1b124, State: Running, Role: LEADER
I20260810 18:01:07.771517 29634 consensus_queue.cc:237] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01cb4e0adb94f91bbbcb0e566c1b124" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39851 } }
I20260810 18:01:07.772344 29629 sys_catalog.cc:564] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:07.772727 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e01cb4e0adb94f91bbbcb0e566c1b124. Latest consensus state: current_term: 1 leader_uuid: "e01cb4e0adb94f91bbbcb0e566c1b124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01cb4e0adb94f91bbbcb0e566c1b124" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39851 } } }
I20260810 18:01:07.772856 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:07.773188 29635 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e01cb4e0adb94f91bbbcb0e566c1b124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01cb4e0adb94f91bbbcb0e566c1b124" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39851 } } }
I20260810 18:01:07.773303 29635 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:07.773670 29638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:07.774501 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:07.760113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.28.83.126:39851)
0810 18:01:07.760579 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:07.760594 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:07.760630 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:01:07.761355 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:07.761373 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:07.761795 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:07.762140 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:07.762983 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:07.764088 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:07.764967 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:07.765535 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:07.765719 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:07.768564 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:07.768570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:07.768593 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:07.768648 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:07.770650 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:07.771202 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:07.771209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:07.771216 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:07.771360 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:07.771780 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:07.771784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:07.771785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:07.772306 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:07.772476 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:07.772991 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 18:01:07.773349 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":132,"threads_started":1}
I20260810 18:01:07.776742 29638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:07.783532 29638 catalog_manager.cc:1347] Generated new cluster ID: e0c7b0b1c0504dc7868f732799471f77
I20260810 18:01:07.783627 29638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:07.797040 29638 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:07.798086 29638 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:07.805190 29638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e01cb4e0adb94f91bbbcb0e566c1b124: Generated new TSK 0
I20260810 18:01:07.806039 29638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:07.836874 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39851
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:07.954975 29656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:07.955320 29656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:07.955403 29656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:07.959613 29656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:07.959684 29656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:07.959779 29656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:07.965372 29656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39851
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:07.966617 29656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:07.967660 29656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:07.977917 29661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:07.977880 29662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:07.978000 29656 server_base.cc:1034] running on GCE node
W20260810 18:01:07.977880 29664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:07.978564 29656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:07.979226 29656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:07.980415 29656 hybrid_clock.cc:648] HybridClock initialized: now 1786384867980380 us; error 48 us; skew 500 ppm
Aug 10 18:01:07 dist-test-slave-srgv krb5kdc[29545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384867, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:07.983592 29656 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:07.984620 29656 webserver.cc:460] Webserver started at http://127.28.83.65:46789/ using document root <none> and password file <none>
I20260810 18:01:07.985211 29656 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:07.985308 29656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:07.985606 29656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:07.987697 29656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "ccf8679d10e1456aa892bb3f61bd2244"
format_stamp: "Formatted at 2026-08-10 18:01:07 on dist-test-slave-srgv"
server_key: "19535887c778d155e3c65691f7677af8"
server_key_iv: "ac65d7f937f82ea5cf56d4be51906558"
server_key_version: "encryptionkey@0"
I20260810 18:01:07.988278 29656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "ccf8679d10e1456aa892bb3f61bd2244"
format_stamp: "Formatted at 2026-08-10 18:01:07 on dist-test-slave-srgv"
server_key: "19535887c778d155e3c65691f7677af8"
server_key_iv: "ac65d7f937f82ea5cf56d4be51906558"
server_key_version: "encryptionkey@0"
I20260810 18:01:07.991953 29656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:01:07.994643 29671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:07.995806 29656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:07.995949 29656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "ccf8679d10e1456aa892bb3f61bd2244"
format_stamp: "Formatted at 2026-08-10 18:01:07 on dist-test-slave-srgv"
server_key: "19535887c778d155e3c65691f7677af8"
server_key_iv: "ac65d7f937f82ea5cf56d4be51906558"
server_key_version: "encryptionkey@0"
I20260810 18:01:07.996057 29656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:08.006120 29656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:08.009522 29656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:08.009724 29656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:08.010305 29656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:08.011108 29656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:08.011185 29656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:08.011257 29656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:08.011301 29656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:08.020054 29656 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:38825
I20260810 18:01:08.020081 29788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:38825 every 8 connection(s)
I20260810 18:01:08.020987 29656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:08.024233 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 29656
I20260810 18:01:08.024346 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:08.024609 29005 external_mini_cluster.cc:1428] Setting key 337972aded52fb7fc9ec7cbbdd4d50d2
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384867, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:01:08.035938 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.022898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:56703 (local address 127.28.83.126:39851)
0810 18:01:08.023058 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.023063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.023903 (+ 840us) server_negotiation.cc:407] Connection header received
0810 18:01:08.025163 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.025167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.025233 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.025314 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:08.026775 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.027291 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.028732 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.028977 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.029097 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.032081 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:08.032085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:08.032087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:08.032111 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:08.033742 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.034267 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.034270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.034272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.034362 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.034761 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.034766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.034768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.034964 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:08.035078 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.035681 (+ 603us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.035808 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:01:08.036936 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.023278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39851 (local address 127.28.83.65:56703)
0810 18:01:08.023735 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:08.023784 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:08.024912 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:08.025545 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:08.025551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:08.025935 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:08.026598 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.026619 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.027420 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.027427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.028574 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.028584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.029142 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.029146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.029313 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.030005 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:08.030020 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:08.031903 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:08.033862 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:08.033868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:08.033889 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:08.034158 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:08.034475 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:08.034478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:08.034480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:08.034662 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:08.035076 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:08.035084 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:08.035350 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 18:01:08.035617 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 18:01:08.035921 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:08.038443 29789 heartbeater.cc:347] Connected to a master server at 127.28.83.126:39851
I20260810 18:01:08.038748 29789 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:08.039340 29789 heartbeater.cc:511] Master 127.28.83.126:39851 requested a full tablet report, sending...
I20260810 18:01:08.040978 29593 ts_manager.cc:194] Registered new tserver with Master: ccf8679d10e1456aa892bb3f61bd2244 (127.28.83.65:38825)
I20260810 18:01:08.042290 29593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:56703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:08.053960 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39851
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:08.169234 29794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:08.169546 29794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:08.169641 29794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:08.173926 29794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:08.173990 29794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:08.174080 29794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:08.179423 29794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39851
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:08.180655 29794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:08.181742 29794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:08.191916 29802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:08.191902 29799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:08.192114 29800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:08.192126 29794 server_base.cc:1034] running on GCE node
I20260810 18:01:08.192684 29794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:08.193338 29794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:08.194504 29794 hybrid_clock.cc:648] HybridClock initialized: now 1786384868194490 us; error 25 us; skew 500 ppm
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384868, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:08.197625 29794 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:08.198513 29794 webserver.cc:460] Webserver started at http://127.28.83.66:44771/ using document root <none> and password file <none>
I20260810 18:01:08.199211 29794 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:08.199281 29794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:08.199530 29794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:08.201726 29794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "c06751442a1f41d08d6a6d914c81c76d"
format_stamp: "Formatted at 2026-08-10 18:01:08 on dist-test-slave-srgv"
server_key: "f9e3391a6b305759029ce0d6b9474603"
server_key_iv: "84f875c5df90b45c1b32a20838293d9b"
server_key_version: "encryptionkey@0"
I20260810 18:01:08.202308 29794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "c06751442a1f41d08d6a6d914c81c76d"
format_stamp: "Formatted at 2026-08-10 18:01:08 on dist-test-slave-srgv"
server_key: "f9e3391a6b305759029ce0d6b9474603"
server_key_iv: "84f875c5df90b45c1b32a20838293d9b"
server_key_version: "encryptionkey@0"
I20260810 18:01:08.206239 29794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:01:08.208739 29809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:08.210017 29794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:08.210175 29794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "c06751442a1f41d08d6a6d914c81c76d"
format_stamp: "Formatted at 2026-08-10 18:01:08 on dist-test-slave-srgv"
server_key: "f9e3391a6b305759029ce0d6b9474603"
server_key_iv: "84f875c5df90b45c1b32a20838293d9b"
server_key_version: "encryptionkey@0"
I20260810 18:01:08.210294 29794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:08.222143 29794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:08.225042 29794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:08.225256 29794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:08.225924 29794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:08.226752 29794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:08.226827 29794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:08.226899 29794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:08.226953 29794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:08.236935 29794 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:43175
I20260810 18:01:08.236999 29927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:43175 every 8 connection(s)
I20260810 18:01:08.237943 29794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:08.241291 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 29794
I20260810 18:01:08.241443 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:08.241752 29005 external_mini_cluster.cc:1428] Setting key d3c91330411a7d7328b6cafc936d6c29
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384868, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
I20260810 18:01:08.253170 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.239874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:35747 (local address 127.28.83.126:39851)
0810 18:01:08.240048 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.240053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.240894 (+ 841us) server_negotiation.cc:407] Connection header received
0810 18:01:08.241951 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.241956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.242014 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.242112 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:08.243614 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.244468 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.246234 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.246457 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.246575 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.249472 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:08.249476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:08.249478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:08.249504 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:08.250983 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.251518 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.251522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.251524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.251607 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.251961 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.251966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.251967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.252224 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:08.252334 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.252850 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.252991 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 18:01:08.254067 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.240273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39851 (local address 127.28.83.66:35747)
0810 18:01:08.240737 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:08.240780 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:08.241685 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:08.242338 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:08.242349 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:08.242869 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:08.243464 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.243478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.244607 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.244610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.246084 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.246099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.246636 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.246641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.246818 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.247482 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:08.247499 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:08.249248 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:08.251113 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:08.251119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:08.251136 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:08.251409 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:08.251711 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:08.251714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:08.251716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:08.251861 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:08.252331 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:08.252339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:08.252600 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 18:01:08.252804 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 18:01:08.253145 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:01:08.255337 29928 heartbeater.cc:347] Connected to a master server at 127.28.83.126:39851
I20260810 18:01:08.255604 29928 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:08.256111 29928 heartbeater.cc:511] Master 127.28.83.126:39851 requested a full tablet report, sending...
I20260810 18:01:08.257180 29593 ts_manager.cc:194] Registered new tserver with Master: c06751442a1f41d08d6a6d914c81c76d (127.28.83.66:43175)
I20260810 18:01:08.257831 29593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:35747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:01:08.271991 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39851
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:08.396142 29933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:08.396486 29933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:08.396581 29933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:08.400976 29933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:08.401094 29933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:08.401191 29933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:08.407196 29933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39851
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:08.408349 29933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:08.409381 29933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:08.419705 29941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:08.419751 29938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:08.419705 29939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:08.420070 29933 server_base.cc:1034] running on GCE node
I20260810 18:01:08.420496 29933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:08.421088 29933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:08.422257 29933 hybrid_clock.cc:648] HybridClock initialized: now 1786384868422239 us; error 26 us; skew 500 ppm
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384868, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:08.425125 29933 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:08.426164 29933 webserver.cc:460] Webserver started at http://127.28.83.67:41271/ using document root <none> and password file <none>
I20260810 18:01:08.426723 29933 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:08.426816 29933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:08.427045 29933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:08.429076 29933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "c6c15628f27141be8fc9c9d18ef13567"
format_stamp: "Formatted at 2026-08-10 18:01:08 on dist-test-slave-srgv"
server_key: "7dceb89c21e73f0f121988640c5ae22f"
server_key_iv: "d0ca7cc6dba09954a6dc807cb62c58c2"
server_key_version: "encryptionkey@0"
I20260810 18:01:08.429690 29933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "c6c15628f27141be8fc9c9d18ef13567"
format_stamp: "Formatted at 2026-08-10 18:01:08 on dist-test-slave-srgv"
server_key: "7dceb89c21e73f0f121988640c5ae22f"
server_key_iv: "d0ca7cc6dba09954a6dc807cb62c58c2"
server_key_version: "encryptionkey@0"
I20260810 18:01:08.433879 29933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:01:08.436532 29948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:08.437923 29933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:08.438087 29933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "c6c15628f27141be8fc9c9d18ef13567"
format_stamp: "Formatted at 2026-08-10 18:01:08 on dist-test-slave-srgv"
server_key: "7dceb89c21e73f0f121988640c5ae22f"
server_key_iv: "d0ca7cc6dba09954a6dc807cb62c58c2"
server_key_version: "encryptionkey@0"
I20260810 18:01:08.438230 29933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:08.461354 29933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:08.464613 29933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:08.464845 29933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:08.465595 29933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:08.466593 29933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:08.466670 29933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:08.466745 29933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:08.466796 29933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:08.485989 29933 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:46737
I20260810 18:01:08.486011 30063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:46737 every 8 connection(s)
I20260810 18:01:08.487011 29933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:08.489650 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 29933
I20260810 18:01:08.489799 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:08.490108 29005 external_mini_cluster.cc:1428] Setting key 57e492b60bcd15253833a24e2670c805
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384868, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:08.502244 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.489026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:45027 (local address 127.28.83.126:39851)
0810 18:01:08.489200 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.489205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.490308 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 18:01:08.491486 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.491491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.491548 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.491636 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:08.493285 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.493932 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.495326 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.495552 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.495652 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.498470 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:08.498473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:08.498475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:08.498500 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:08.500003 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.500552 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.500555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.500556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.500660 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.500960 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.500962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.500964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.501164 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:08.501241 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.501997 (+ 756us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.502124 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:01:08.503204 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.489398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39851 (local address 127.28.83.67:45027)
0810 18:01:08.490102 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:08.490174 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:08.491309 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:08.491832 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:08.491841 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:08.492481 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:08.493106 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.493148 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.494040 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.494042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.495218 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.495228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.495694 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.495697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.495842 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.496533 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:08.496550 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:08.498318 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:08.500109 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:08.500114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:08.500125 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:08.500458 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:08.500747 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:08.500749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:08.500751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:08.500882 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:08.501471 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:08.501480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:08.501747 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 18:01:08.501961 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 18:01:08.502233 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":140,"threads_started":1}
I20260810 18:01:08.504431 30064 heartbeater.cc:347] Connected to a master server at 127.28.83.126:39851
I20260810 18:01:08.504679 30064 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:08.505174 30064 heartbeater.cc:511] Master 127.28.83.126:39851 requested a full tablet report, sending...
I20260810 18:01:08.506270 29593 ts_manager.cc:194] Registered new tserver with Master: c6c15628f27141be8fc9c9d18ef13567 (127.28.83.67:46737)
I20260810 18:01:08.506825 29593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:45027
I20260810 18:01:08.515882 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:01:08.528195 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.520178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.28.83.126:39851)
0810 18:01:08.520314 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.520318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.520598 (+ 280us) server_negotiation.cc:407] Connection header received
0810 18:01:08.520726 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.520729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.520771 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.520846 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:08.521733 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.522603 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.523405 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.523612 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.523686 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.525047 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:08.525049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:08.525051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:08.525075 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:08.526663 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.527046 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.527049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.527051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.527128 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.527356 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.527358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.527359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.527582 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:08.527686 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.527878 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.527978 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:01:08.533608 29593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:08.536957 29593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:08.554174 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.547147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.28.83.67:46737)
0810 18:01:08.548361 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.548375 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.548414 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:01:08.548514 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.548521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.548773 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.548960 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:08.549361 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.550726 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.552566 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.553271 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.553407 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.553584 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.553938 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.554035 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":162,"threads_started":1}
I20260810 18:01:08.554152 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.547888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.28.83.66:43175)
0810 18:01:08.548592 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.548633 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.548654 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:01:08.548721 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.548725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.548898 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.549480 (+ 582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:08.549704 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.550521 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.552512 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.553316 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.553660 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.553805 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.553862 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.553965 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":180,"threads_started":1}
I20260810 18:01:08.554210 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.547887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.28.83.65:38825)
0810 18:01:08.548486 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.548494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.548521 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:01:08.548606 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.548613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.548839 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.549204 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:08.549433 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.550719 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.552279 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.553168 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.553288 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.553559 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.553913 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.554003 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":92,"threads_started":1}
I20260810 18:01:08.554147 30079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.547626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:38825 (local address 127.0.0.1:45812)
0810 18:01:08.548111 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:08.548126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:08.548263 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:08.549007 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:08.549010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:08.549021 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:08.549292 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.549301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.550911 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.550916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.552135 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.552146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.553618 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.553624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.553727 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.553749 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:01:08.553828 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:01:08.553964 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":171,"spinlock_wait_cycles":4096,"thread_start_us":213,"threads_started":1}
I20260810 18:01:08.555080 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.547142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:43175 (local address 127.0.0.1:36900)
0810 18:01:08.547621 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:08.547639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:08.547744 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:08.549113 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:08.549119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:08.549143 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:08.549305 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.549310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.550752 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.550758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.552365 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.552377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.553507 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.553511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.553608 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.553725 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 18:01:08.553785 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 18:01:08.554917 (+ 1132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":1229,"thread_start_us":165,"threads_started":1}
I20260810 18:01:08.555944 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.546830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:46737 (local address 127.0.0.1:56214)
0810 18:01:08.547336 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:08.547388 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:08.547523 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:08.548958 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:08.548967 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:08.548995 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:08.549216 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.549225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.550959 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.550962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.552427 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.552438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.553431 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.553435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.553575 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.553652 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 18:01:08.553779 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 18:01:08.554889 (+ 1110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":1072,"spinlock_wait_cycles":1920,"thread_start_us":108,"threads_started":1}
I20260810 18:01:08.558184 29720 tablet_service.cc:1467] Processing CreateTablet for tablet dbcbf9adc51d4e2cb8e5ee7fb0d74e26 (DEFAULT_TABLE table=test-table [id=56075c65f6f7423a94b3934c77e037b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:08.559541 29720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbcbf9adc51d4e2cb8e5ee7fb0d74e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:08.560330 29997 tablet_service.cc:1467] Processing CreateTablet for tablet dbcbf9adc51d4e2cb8e5ee7fb0d74e26 (DEFAULT_TABLE table=test-table [id=56075c65f6f7423a94b3934c77e037b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:08.560607 29859 tablet_service.cc:1467] Processing CreateTablet for tablet dbcbf9adc51d4e2cb8e5ee7fb0d74e26 (DEFAULT_TABLE table=test-table [id=56075c65f6f7423a94b3934c77e037b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:08.561460 29997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbcbf9adc51d4e2cb8e5ee7fb0d74e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:08.561661 29859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbcbf9adc51d4e2cb8e5ee7fb0d74e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:08.568054 30082 tablet_bootstrap.cc:492] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d: Bootstrap starting.
I20260810 18:01:08.569736 30083 tablet_bootstrap.cc:492] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244: Bootstrap starting.
I20260810 18:01:08.570515 30084 tablet_bootstrap.cc:492] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567: Bootstrap starting.
I20260810 18:01:08.571931 30082 tablet_bootstrap.cc:654] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:08.572782 30083 tablet_bootstrap.cc:654] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:08.573312 30084 tablet_bootstrap.cc:654] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:08.573397 30082 log.cc:824] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:08.574119 30083 log.cc:824] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:08.574841 30084 log.cc:824] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:08.575944 30082 tablet_bootstrap.cc:492] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d: No bootstrap required, opened a new log
I20260810 18:01:08.576170 30082 ts_tablet_manager.cc:1397] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 18:01:08.576397 30083 tablet_bootstrap.cc:492] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244: No bootstrap required, opened a new log
I20260810 18:01:08.576588 30083 ts_tablet_manager.cc:1397] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:01:08.579532 30084 tablet_bootstrap.cc:492] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567: No bootstrap required, opened a new log
I20260810 18:01:08.579836 30084 ts_tablet_manager.cc:1397] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 18:01:08.581364 30082 raft_consensus.cc:348] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } }
I20260810 18:01:08.581821 30082 raft_consensus.cc:374] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:08.581895 30082 raft_consensus.cc:729] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06751442a1f41d08d6a6d914c81c76d, State: Initialized, Role: FOLLOWER
I20260810 18:01:08.582185 30083 raft_consensus.cc:348] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } }
I20260810 18:01:08.582425 30083 raft_consensus.cc:374] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:08.582383 30082 consensus_queue.cc:260] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } }
I20260810 18:01:08.582479 30083 raft_consensus.cc:729] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf8679d10e1456aa892bb3f61bd2244, State: Initialized, Role: FOLLOWER
I20260810 18:01:08.582922 30083 consensus_queue.cc:260] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } }
I20260810 18:01:08.583400 29928 heartbeater.cc:503] Master 127.28.83.126:39851 was elected leader, sending a full tablet report...
I20260810 18:01:08.583556 30082 ts_tablet_manager.cc:1428] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260810 18:01:08.583958 30083 ts_tablet_manager.cc:1428] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 18:01:08.584208 29789 heartbeater.cc:503] Master 127.28.83.126:39851 was elected leader, sending a full tablet report...
I20260810 18:01:08.584014 30084 raft_consensus.cc:348] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } }
I20260810 18:01:08.585021 30084 raft_consensus.cc:374] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:08.585099 30084 raft_consensus.cc:729] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c15628f27141be8fc9c9d18ef13567, State: Initialized, Role: FOLLOWER
I20260810 18:01:08.585558 30084 consensus_queue.cc:260] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } }
I20260810 18:01:08.586325 30064 heartbeater.cc:503] Master 127.28.83.126:39851 was elected leader, sending a full tablet report...
I20260810 18:01:08.586704 30084 ts_tablet_manager.cc:1428] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:01:08.596563 30090 raft_consensus.cc:482] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:08.596781 30090 raft_consensus.cc:504] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } }
I20260810 18:01:08.598112 30090 leader_election.cc:288] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c06751442a1f41d08d6a6d914c81c76d (127.28.83.66:43175), ccf8679d10e1456aa892bb3f61bd2244 (127.28.83.65:38825)
I20260810 18:01:08.602553 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.598473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:41433 (local address 127.28.83.66:43175)
0810 18:01:08.598619 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.598624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.598644 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:01:08.598727 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.598730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.598786 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.598920 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:08.599307 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.600139 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.601301 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.601976 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.602087 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.602142 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.602397 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.602450 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:01:08.602586 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.598337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:43175 (local address 127.28.83.67:41433)
0810 18:01:08.598481 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:08.598497 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:08.598614 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:08.598960 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:08.598963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:08.598981 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:08.599169 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.599178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.600258 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.600260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.601175 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.601184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.602110 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.602114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.602247 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.602292 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 18:01:08.602375 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 18:01:08.602453 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 18:01:08.603318 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.598721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:43587 (local address 127.28.83.65:38825)
0810 18:01:08.598848 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.598853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.599099 (+ 246us) server_negotiation.cc:407] Connection header received
0810 18:01:08.599209 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.599214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.599277 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.599357 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:08.599840 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.600468 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.601911 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.602769 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.602891 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.602941 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.603136 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.603205 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:01:08.603330 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.598457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:38825 (local address 127.28.83.67:43587)
0810 18:01:08.598983 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:08.598996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:08.599115 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:08.599387 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:08.599390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:08.599409 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:08.599645 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.599653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.600579 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.600581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.601794 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:08.601805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.602912 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.602916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:08.603018 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.603034 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 18:01:08.603127 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 18:01:08.603192 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":212,"threads_started":1}
I20260810 18:01:08.603322 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcbf9adc51d4e2cb8e5ee7fb0d74e26" candidate_uuid: "c6c15628f27141be8fc9c9d18ef13567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06751442a1f41d08d6a6d914c81c76d" is_pre_election: true
I20260810 18:01:08.603796 29879 raft_consensus.cc:2454] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c15628f27141be8fc9c9d18ef13567 in term 0.
I20260810 18:01:08.603858 29741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcbf9adc51d4e2cb8e5ee7fb0d74e26" candidate_uuid: "c6c15628f27141be8fc9c9d18ef13567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf8679d10e1456aa892bb3f61bd2244" is_pre_election: true
I20260810 18:01:08.604146 29741 raft_consensus.cc:2454] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c15628f27141be8fc9c9d18ef13567 in term 0.
I20260810 18:01:08.604410 29950 leader_election.cc:302] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c06751442a1f41d08d6a6d914c81c76d, c6c15628f27141be8fc9c9d18ef13567; no voters:
I20260810 18:01:08.604697 30090 raft_consensus.cc:2781] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:08.604786 30090 raft_consensus.cc:482] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:08.604830 30090 raft_consensus.cc:3037] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:08.606263 30090 raft_consensus.cc:504] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } }
I20260810 18:01:08.606618 30090 leader_election.cc:288] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [CANDIDATE]: Term 1 election: Requested vote from peers c06751442a1f41d08d6a6d914c81c76d (127.28.83.66:43175), ccf8679d10e1456aa892bb3f61bd2244 (127.28.83.65:38825)
I20260810 18:01:08.607069 29741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcbf9adc51d4e2cb8e5ee7fb0d74e26" candidate_uuid: "c6c15628f27141be8fc9c9d18ef13567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf8679d10e1456aa892bb3f61bd2244"
I20260810 18:01:08.607154 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbcbf9adc51d4e2cb8e5ee7fb0d74e26" candidate_uuid: "c6c15628f27141be8fc9c9d18ef13567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06751442a1f41d08d6a6d914c81c76d"
I20260810 18:01:08.607225 29741 raft_consensus.cc:3037] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:08.607300 29879 raft_consensus.cc:3037] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:08.608867 29741 raft_consensus.cc:2454] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P ccf8679d10e1456aa892bb3f61bd2244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c15628f27141be8fc9c9d18ef13567 in term 1.
I20260810 18:01:08.608918 29879 raft_consensus.cc:2454] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c06751442a1f41d08d6a6d914c81c76d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c15628f27141be8fc9c9d18ef13567 in term 1.
I20260810 18:01:08.609261 29949 leader_election.cc:302] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6c15628f27141be8fc9c9d18ef13567, ccf8679d10e1456aa892bb3f61bd2244; no voters:
I20260810 18:01:08.609526 30090 raft_consensus.cc:2781] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:08.609769 30090 raft_consensus.cc:686] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [term 1 LEADER]: Becoming Leader. State: Replica: c6c15628f27141be8fc9c9d18ef13567, State: Running, Role: LEADER
I20260810 18:01:08.610102 30090 consensus_queue.cc:237] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } }
I20260810 18:01:08.613620 29591 catalog_manager.cc:5495] T dbcbf9adc51d4e2cb8e5ee7fb0d74e26 P c6c15628f27141be8fc9c9d18ef13567 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c15628f27141be8fc9c9d18ef13567 (127.28.83.67). New cstate: current_term: 1 leader_uuid: "c6c15628f27141be8fc9c9d18ef13567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c15628f27141be8fc9c9d18ef13567" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c06751442a1f41d08d6a6d914c81c76d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf8679d10e1456aa892bb3f61bd2244" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 38825 } health_report { overall_health: UNKNOWN } } }
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:08.640441 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.630302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.28.83.65:38825)
0810 18:01:08.630475 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.630481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.630616 (+ 135us) server_negotiation.cc:407] Connection header received
0810 18:01:08.630726 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.630729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.630773 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.630848 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:08.631581 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.632081 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.633109 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.633560 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.633652 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.636459 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:08.636474 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:08.636490 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:08.636518 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:08.638366 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.638839 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.638846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.638848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.638988 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.639380 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.639384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.639386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.639686 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:08.639941 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.640134 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.640271 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:08.670931 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.661489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45828 (local address 127.28.83.65:38825)
0810 18:01:08.661676 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.661679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.661793 (+ 114us) server_negotiation.cc:407] Connection header received
0810 18:01:08.661920 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.661922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.661968 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.662040 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:08.662750 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.663271 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.664707 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.665018 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.665109 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.667718 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:08.667722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:08.667724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:08.667751 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:08.669238 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.669753 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.669756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.669757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.669837 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.670098 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.670100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.670101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.670274 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:08.670390 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.670689 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.670804 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[29545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:08.701920 30080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:08.692582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.28.83.65:38825)
0810 18:01:08.692751 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:08.692755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:08.692901 (+ 146us) server_negotiation.cc:407] Connection header received
0810 18:01:08.693022 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:08.693025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:08.693072 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:08.693163 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:08.694041 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.694597 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.695422 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:08.695661 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:08.695744 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:08.698448 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:08.698452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:08.698454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:08.698481 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:08.700177 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.700765 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.700768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.700769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.700868 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:08.701169 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:08.701171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:08.701172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:08.701339 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:08.701500 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:08.701691 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 18:01:08.701796 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 18:01:08.703496 29699 tablet_service.cc:2908] Rejecting scan request for tablet dbcbf9adc51d4e2cb8e5ee7fb0d74e26: Uninitialized: clean time has not yet been initialized
I20260810 18:01:08.706553 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 29656
I20260810 18:01:08.713186 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 29794
I20260810 18:01:08.719375 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 29933
I20260810 18:01:08.725704 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 29558
2026-08-10T18:01:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2358 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[30106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[30106](info): set up 2 sockets
Aug 10 18:01:08 dist-test-slave-srgv krb5kdc[30106](info): commencing operation
krb5kdc: starting...
W20260810 18:01:09.789238 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:09 dist-test-slave-srgv krb5kdc[30106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:09.918848 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:42251
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:42251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:10.036652 30119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:10.037001 30119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:10.037083 30119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:10.041250 30119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:10.041383 30119 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:01:10.041416 30119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:10.041472 30119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:10.041512 30119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:10.041549 30119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:10.047456 30119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:42251
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:10.048696 30119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:10.049831 30119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:10.058831 30128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:10.058837 30125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:10.058897 30126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:10.059319 30119 server_base.cc:1034] running on GCE node
I20260810 18:01:10.059965 30119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:10.061177 30119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:10.062386 30119 hybrid_clock.cc:648] HybridClock initialized: now 1786384870062390 us; error 58 us; skew 500 ppm
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:10.065390 30119 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:10.066494 30119 webserver.cc:460] Webserver started at http://127.28.83.126:46695/ using document root <none> and password file <none>
I20260810 18:01:10.067108 30119 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:10.067211 30119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:10.067459 30119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:10.070174 30119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "fba2bcf92e364250bfd9402a1af5d905"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "dbf3dabf11124108051db42cdd6e07e7"
server_key_iv: "bf5b6c51f42c9caf9a44dbb4711f00c7"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.070819 30119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "fba2bcf92e364250bfd9402a1af5d905"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "dbf3dabf11124108051db42cdd6e07e7"
server_key_iv: "bf5b6c51f42c9caf9a44dbb4711f00c7"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.075071 30119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:01:10.077869 30136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:10.079275 30119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:10.079473 30119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "fba2bcf92e364250bfd9402a1af5d905"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "dbf3dabf11124108051db42cdd6e07e7"
server_key_iv: "bf5b6c51f42c9caf9a44dbb4711f00c7"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.079623 30119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:10.117709 30119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:10.120668 30119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:10.120895 30119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:10.129776 30119 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:42251
I20260810 18:01:10.129813 30200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:42251 every 8 connection(s)
I20260810 18:01:10.130847 30119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:10.133677 30201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:10.136798 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 30119
I20260810 18:01:10.136936 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:10.137214 29005 external_mini_cluster.cc:1428] Setting key f1d9f0953b386b222f379e06f7442dcd
I20260810 18:01:10.139950 30201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905: Bootstrap starting.
I20260810 18:01:10.142635 30201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:10.143587 30201 log.cc:824] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:10.146173 30201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905: No bootstrap required, opened a new log
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:10.149921 30201 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba2bcf92e364250bfd9402a1af5d905" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42251 } }
I20260810 18:01:10.150213 30201 raft_consensus.cc:374] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:10.150282 30201 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba2bcf92e364250bfd9402a1af5d905, State: Initialized, Role: FOLLOWER
I20260810 18:01:10.150826 30201 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba2bcf92e364250bfd9402a1af5d905" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42251 } }
I20260810 18:01:10.151005 30201 raft_consensus.cc:388] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:10.151075 30201 raft_consensus.cc:482] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:10.151171 30201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:10.152453 30201 raft_consensus.cc:504] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba2bcf92e364250bfd9402a1af5d905" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42251 } }
I20260810 18:01:10.152835 30201 leader_election.cc:302] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fba2bcf92e364250bfd9402a1af5d905; no voters:
I20260810 18:01:10.153095 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.138735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.28.83.126:42251)
0810 18:01:10.139179 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.139195 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.139233 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:01:10.139846 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.139864 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.140266 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.140557 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:10.141593 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.142606 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.143817 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.144485 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.144631 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.147582 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:10.147587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:10.147601 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:10.147630 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:10.149547 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.150027 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.150040 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.150050 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.150171 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.150631 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.150634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.150636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.151229 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:10.151467 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.151670 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.151988 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260810 18:01:10.153198 30201 leader_election.cc:288] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:10.153479 30206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:10.153738 30206 raft_consensus.cc:686] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [term 1 LEADER]: Becoming Leader. State: Replica: fba2bcf92e364250bfd9402a1af5d905, State: Running, Role: LEADER
I20260810 18:01:10.154074 30206 consensus_queue.cc:237] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba2bcf92e364250bfd9402a1af5d905" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42251 } }
I20260810 18:01:10.154537 30201 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:10.155682 30207 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fba2bcf92e364250bfd9402a1af5d905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba2bcf92e364250bfd9402a1af5d905" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42251 } } }
I20260810 18:01:10.155814 30207 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:10.156260 30208 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fba2bcf92e364250bfd9402a1af5d905. Latest consensus state: current_term: 1 leader_uuid: "fba2bcf92e364250bfd9402a1af5d905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba2bcf92e364250bfd9402a1af5d905" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42251 } } }
I20260810 18:01:10.156308 30215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:10.156431 30208 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:10.159684 30215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:10.165932 30215 catalog_manager.cc:1347] Generated new cluster ID: 80b0833ca12748dfa59006dbc5f8aa77
I20260810 18:01:10.166018 30215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:10.197268 30215 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:10.198649 30215 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:10.209457 30215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905: Generated new TSK 0
I20260810 18:01:10.210449 30215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:10.242653 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42251
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:10.360450 30227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:10.360740 30227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:10.360790 30227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:10.364876 30227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:10.364948 30227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:10.364989 30227 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:01:10.365023 30227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:10.370604 30227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.83.126:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:10.371711 30227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:10.372892 30227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:10.382968 30233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:10.382968 30232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:10.383298 30227 server_base.cc:1034] running on GCE node
W20260810 18:01:10.383014 30235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:10.383909 30227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:10.384635 30227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:10.385834 30227 hybrid_clock.cc:648] HybridClock initialized: now 1786384870385821 us; error 27 us; skew 500 ppm
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:10.388767 30227 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:10.389837 30227 webserver.cc:460] Webserver started at http://127.28.83.65:38945/ using document root <none> and password file <none>
I20260810 18:01:10.390419 30227 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:10.390515 30227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:10.390766 30227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:10.392860 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "49bc2c83960241fa8868978f3b22a011"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "fe878be0465650219d97a6c5a769ba0d"
server_key_iv: "297f1bf87e39c73c739c2ad52918ce78"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.393708 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "49bc2c83960241fa8868978f3b22a011"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "fe878be0465650219d97a6c5a769ba0d"
server_key_iv: "297f1bf87e39c73c739c2ad52918ce78"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.397500 30227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:01:10.399956 30244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:10.401245 30227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:01:10.401388 30227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "49bc2c83960241fa8868978f3b22a011"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "fe878be0465650219d97a6c5a769ba0d"
server_key_iv: "297f1bf87e39c73c739c2ad52918ce78"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.401564 30227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:10.418073 30227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:10.421126 30227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:10.421358 30227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:10.422889 30227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:10.422969 30227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:10.423043 30227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:10.423092 30227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:10.434504 30227 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:39931
I20260810 18:01:10.434588 30365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:39931 every 8 connection(s)
I20260810 18:01:10.435773 30227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:10.438880 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.423785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:47929 (local address 127.28.83.126:42251)
0810 18:01:10.423945 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.423950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.424914 (+ 964us) server_negotiation.cc:407] Connection header received
0810 18:01:10.426011 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.426015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.426073 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.426138 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:10.428047 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.428940 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.430955 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.431430 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.431542 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.434807 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:10.434812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:10.434815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:10.434848 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:10.436641 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.437213 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.437218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.437219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.437303 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.437719 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.437723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.437724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.437927 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:10.438039 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.438555 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.438697 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:01:10.439677 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.424216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42251 (local address 127.28.83.65:47929)
0810 18:01:10.424676 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:10.424738 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:10.425745 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:10.426591 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:10.426600 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:10.427107 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:10.427846 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.427868 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.429131 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.429136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.430792 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.430809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.431651 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.431656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.431906 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.432729 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:10.432746 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:10.434540 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:10.436780 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.436786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.436802 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.437077 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.437401 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.437430 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.437433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.437610 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.438039 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:10.438047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:10.438305 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 18:01:10.438495 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 18:01:10.438848 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":133,"threads_started":1}
I20260810 18:01:10.440706 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 30227
I20260810 18:01:10.440848 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:10.441085 30366 heartbeater.cc:347] Connected to a master server at 127.28.83.126:42251
I20260810 18:01:10.441128 29005 external_mini_cluster.cc:1428] Setting key d4ada1ca6c7c7a0bb7bd8cef8d439027
I20260810 18:01:10.441332 30366 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:10.441994 30366 heartbeater.cc:511] Master 127.28.83.126:42251 requested a full tablet report, sending...
I20260810 18:01:10.443738 30153 ts_manager.cc:194] Registered new tserver with Master: 49bc2c83960241fa8868978f3b22a011 (127.28.83.65:39931)
I20260810 18:01:10.445468 30153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:47929
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:01:10.453038 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.444459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:52371 (local address 127.28.83.126:42251)
0810 18:01:10.444607 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.444614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.444949 (+ 335us) server_negotiation.cc:407] Connection header received
0810 18:01:10.445100 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.445104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.445150 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.445239 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:10.445833 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.446478 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.447854 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.448071 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.448157 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.449664 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:10.449667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:10.449679 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:10.449702 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:10.451216 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.451766 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.451769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.451770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.451843 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.452168 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.452171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.452173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.452347 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:10.452422 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.452790 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.452906 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:01:10.453069 30373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.443776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42251 (local address 127.28.83.65:52371)
0810 18:01:10.444847 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:10.444861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:10.444994 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:10.445257 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:10.445259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:10.445551 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:10.445708 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.445716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.446626 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.446632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.447690 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.447718 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.448208 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.448214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.448333 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.448901 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:10.448922 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:10.449516 (+ 594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:10.451350 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.451354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.451357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.451643 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.451944 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.451950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.451954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.452063 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.452445 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:10.452450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:10.452579 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 18:01:10.452719 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 18:01:10.452915 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":883,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:10.472747 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42251
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:10.587774 30375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:10.588047 30375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:10.588129 30375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:10.592252 30375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:10.592315 30375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:10.592394 30375 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:01:10.592483 30375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:10.597925 30375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.83.126:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:10.599016 30375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:10.600077 30375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:01:10.610023 30375 server_base.cc:1034] running on GCE node
W20260810 18:01:10.609984 30383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:10.609942 30381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:10.609970 30380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:10.610652 30375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:10.611304 30375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:10.612479 30375 hybrid_clock.cc:648] HybridClock initialized: now 1786384870612458 us; error 33 us; skew 500 ppm
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:10.615422 30375 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:10.616343 30375 webserver.cc:460] Webserver started at http://127.28.83.66:43751/ using document root <none> and password file <none>
I20260810 18:01:10.616909 30375 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:10.616972 30375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:10.617465 30375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:10.619583 30375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "0c89f94e1d7c4b53b651eae6b870a708"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "32e36a6456aeabcc91eb0229379d27c3"
server_key_iv: "04cd630dfb45728c8fbb94eea16c920a"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.620230 30375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "0c89f94e1d7c4b53b651eae6b870a708"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "32e36a6456aeabcc91eb0229379d27c3"
server_key_iv: "04cd630dfb45728c8fbb94eea16c920a"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.624123 30375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:01:10.626734 30390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:10.627956 30375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:10.628077 30375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "0c89f94e1d7c4b53b651eae6b870a708"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "32e36a6456aeabcc91eb0229379d27c3"
server_key_iv: "04cd630dfb45728c8fbb94eea16c920a"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.628187 30375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:10.644008 30375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:10.647464 30375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:10.647629 30375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:10.649112 30375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:10.649194 30375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:10.649276 30375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:10.649333 30375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:10.660266 30375 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:41157
I20260810 18:01:10.660342 30509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:41157 every 8 connection(s)
I20260810 18:01:10.661350 30375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:10.664724 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.650068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:33731 (local address 127.28.83.126:42251)
0810 18:01:10.650258 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.650266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.651062 (+ 796us) server_negotiation.cc:407] Connection header received
0810 18:01:10.652617 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.652623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.652692 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.652814 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:10.654458 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.655385 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.657160 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.657545 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.657668 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.660920 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:10.660923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:10.660925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:10.660951 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:10.662630 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.663174 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.663177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.663179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.663268 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.663615 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.663620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.663621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.663807 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:10.663917 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.664432 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.664552 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:01:10.665553 30401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.650441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42251 (local address 127.28.83.66:33731)
0810 18:01:10.650894 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:10.650944 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:10.652356 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:10.653012 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:10.653019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:10.653575 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:10.654311 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.654333 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.655513 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.655519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.657023 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.657034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.657715 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.657719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.657894 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.658609 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:10.658640 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:10.660751 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:10.662755 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.662761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.662776 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.663061 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.663372 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.663375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.663377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.663516 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.663915 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:10.663922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:10.664196 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 18:01:10.664393 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 18:01:10.664716 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
I20260810 18:01:10.666675 30510 heartbeater.cc:347] Connected to a master server at 127.28.83.126:42251
I20260810 18:01:10.666814 30510 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:10.667395 30510 heartbeater.cc:511] Master 127.28.83.126:42251 requested a full tablet report, sending...
I20260810 18:01:10.668481 30149 ts_manager.cc:194] Registered new tserver with Master: 0c89f94e1d7c4b53b651eae6b870a708 (127.28.83.66:41157)
I20260810 18:01:10.669284 30149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:33731
I20260810 18:01:10.669886 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 30375
I20260810 18:01:10.670015 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:10.670323 29005 external_mini_cluster.cc:1428] Setting key 18c9404e7c8481e6bbc128031db70de9
I20260810 18:01:10.677217 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.669026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42251 (local address 127.28.83.66:60037)
0810 18:01:10.669321 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:10.669335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:10.669480 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:10.669828 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:10.669831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:10.670073 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:10.670215 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.670222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.671110 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.671114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.671909 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.671920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.672350 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.672356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.672490 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.673009 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:10.673028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:10.673531 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:10.675588 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.675590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.675592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.675786 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.676107 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.676109 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.676110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.676183 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.676743 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:10.676747 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:10.676854 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 18:01:10.676939 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:01:10.677058 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260810 18:01:10.677356 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.669171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:60037 (local address 127.28.83.126:42251)
0810 18:01:10.669294 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.669299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.669469 (+ 170us) server_negotiation.cc:407] Connection header received
0810 18:01:10.669569 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.669572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.669639 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.669718 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:10.670329 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.670981 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.672023 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.672232 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.672310 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.673681 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:10.673687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:10.673693 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:10.673725 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:10.675470 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.675883 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.675890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.675894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.675991 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.676286 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.676292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.676297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.676629 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:10.676747 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.677047 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.677176 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:01:10.700246 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42251
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:10.815662 30519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:10.816009 30519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:10.816097 30519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:10.820169 30519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:10.820266 30519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:10.820351 30519 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:01:10.820430 30519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:10.825937 30519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.83.126:42251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:10.827135 30519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:10.828181 30519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:10.838058 30524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:10.838065 30527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:10.838176 30525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:10.838209 30519 server_base.cc:1034] running on GCE node
I20260810 18:01:10.838748 30519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:10.839438 30519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:10.840636 30519 hybrid_clock.cc:648] HybridClock initialized: now 1786384870840621 us; error 34 us; skew 500 ppm
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:10.843681 30519 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:10.844658 30519 webserver.cc:460] Webserver started at http://127.28.83.67:42081/ using document root <none> and password file <none>
I20260810 18:01:10.845252 30519 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:10.845350 30519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:10.845605 30519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:10.847703 30519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "0a407492eff84a138b1a88deff8a2dbe"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "a9f8bf02c56bc7b286c55e842413ecd8"
server_key_iv: "63933d4f61619147a14c5754b70bc909"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.848313 30519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "0a407492eff84a138b1a88deff8a2dbe"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "a9f8bf02c56bc7b286c55e842413ecd8"
server_key_iv: "63933d4f61619147a14c5754b70bc909"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.852208 30519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:01:10.854902 30535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:10.856381 30519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:01:10.856518 30519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "0a407492eff84a138b1a88deff8a2dbe"
format_stamp: "Formatted at 2026-08-10 18:01:10 on dist-test-slave-srgv"
server_key: "a9f8bf02c56bc7b286c55e842413ecd8"
server_key_iv: "63933d4f61619147a14c5754b70bc909"
server_key_version: "encryptionkey@0"
I20260810 18:01:10.856647 30519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:10.875728 30519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:10.878804 30519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:10.879014 30519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:10.880450 30519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:10.880498 30519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:10.880564 30519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:10.880596 30519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:10.891824 30519 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:33727
I20260810 18:01:10.892035 30658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:33727 every 8 connection(s)
I20260810 18:01:10.892972 30519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:10.896091 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.881534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:53019 (local address 127.28.83.126:42251)
0810 18:01:10.881672 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.881676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.882765 (+ 1089us) server_negotiation.cc:407] Connection header received
0810 18:01:10.883994 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.883998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.884091 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.884215 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:10.885987 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.886896 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.888795 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.889022 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.889116 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.892427 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:10.892436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:10.892439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:10.892468 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:10.894239 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.894819 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.894821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.894823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.894899 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.895189 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.895192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.895193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.895364 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:10.895438 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.895863 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.895963 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:01:10.896963 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.882104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42251 (local address 127.28.83.67:53019)
0810 18:01:10.882610 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:10.882654 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:10.883702 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:10.884449 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:10.884461 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:10.884928 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:10.885813 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.885840 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.887052 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.887058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.888650 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.888663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.889227 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.889232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.889482 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.890227 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:10.890251 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:10.892255 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:10.894340 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.894344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.894379 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.894719 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.894976 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.894978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.894979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.895104 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.895448 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:10.895453 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:10.895661 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 18:01:10.895802 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 18:01:10.896033 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":132,"threads_started":1}
I20260810 18:01:10.897176 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 30519
I20260810 18:01:10.897287 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:10.897636 29005 external_mini_cluster.cc:1428] Setting key 83d29528ef41ed98acef74ae0e39c6f2
I20260810 18:01:10.898342 30659 heartbeater.cc:347] Connected to a master server at 127.28.83.126:42251
I20260810 18:01:10.898490 30659 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:10.899052 30659 heartbeater.cc:511] Master 127.28.83.126:42251 requested a full tablet report, sending...
I20260810 18:01:10.900071 30153 ts_manager.cc:194] Registered new tserver with Master: 0a407492eff84a138b1a88deff8a2dbe (127.28.83.67:33727)
I20260810 18:01:10.900822 30153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:53019
I20260810 18:01:10.907320 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.900540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:55575 (local address 127.28.83.126:42251)
0810 18:01:10.900667 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.900676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.900886 (+ 210us) server_negotiation.cc:407] Connection header received
0810 18:01:10.901002 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.901005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.901052 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.901167 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:10.901701 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.902199 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.903064 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.903277 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.903385 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.904399 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:10.904402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:10.904405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:10.904428 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:10.905765 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.906160 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.906165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.906167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.906249 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.906526 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.906531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.906532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.906698 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:10.906804 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.907075 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.907183 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:01:10.907330 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.900456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42251 (local address 127.28.83.67:55575)
0810 18:01:10.900803 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:10.900815 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:10.900927 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:10.901183 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:10.901186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:10.901381 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:10.901589 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.901595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.902349 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.902352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.902932 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.902940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.903390 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.903394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.903494 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.903948 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:10.903958 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:10.904266 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:10.905862 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.905865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.905867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.906062 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.906343 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:10.906346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:10.906348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:10.906435 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:10.906795 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:10.906798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:10.906900 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 18:01:10.907008 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 18:01:10.907146 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":18,"thread_start_us":112,"threads_started":1}
I20260810 18:01:10.911378 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:01:10 dist-test-slave-srgv krb5kdc[30106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:10.937559 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.929128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.28.83.126:42251)
0810 18:01:10.929265 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.929268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.929280 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:01:10.929321 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.929323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.929386 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.929532 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:10.930263 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.930808 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.931782 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.931999 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.932109 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.934697 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:10.934700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:10.934701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:10.934728 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:10.935982 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.936373 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.936376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.936377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.936454 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:10.936682 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:10.936685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:10.936686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:10.936852 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:10.936950 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.937214 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.937317 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:01:10.940469 30153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:10.943666 30153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:10.958694 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.952507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41157 (local address 127.0.0.1:34352)
0810 18:01:10.953244 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:10.953258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:10.953349 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:10.954189 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:10.954192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:10.954210 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:10.954563 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.954571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.956042 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.956045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.957221 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.957232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.958375 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.958380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.958460 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.958480 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:01:10.958538 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 18:01:10.958590 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":239,"thread_start_us":107,"threads_started":1}
I20260810 18:01:10.958812 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.953089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.28.83.66:41157)
0810 18:01:10.953791 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.953802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.953822 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:01:10.953884 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.953888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.954038 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.954326 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:10.954686 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.955878 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.957353 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.958196 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.958352 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.958520 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.958572 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.958658 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":122,"threads_started":1}
I20260810 18:01:10.958688 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.952041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:33727 (local address 127.0.0.1:55210)
0810 18:01:10.952414 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:10.952469 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:10.952623 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:10.954328 (+ 1705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:10.954331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:10.954345 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:10.954444 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.954450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.955645 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.955652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.957161 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.957172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.958085 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.958088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.958204 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.958243 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 18:01:10.958385 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 18:01:10.958511 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260810 18:01:10.959316 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.952212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.28.83.67:33727)
0810 18:01:10.953865 (+ 1653us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.953875 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.953892 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:01:10.953951 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.953955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.954146 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.954341 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:10.954585 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.955503 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.957307 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.957952 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.958085 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.958265 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.958927 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.959026 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1562,"thread_start_us":106,"threads_started":1}
I20260810 18:01:10.960120 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.952857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:39931 (local address 127.0.0.1:55728)
0810 18:01:10.953544 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:10.953554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:10.953636 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:10.954127 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:10.954133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:10.954154 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:10.954346 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.954356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.956299 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.956302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.957278 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:10.957291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.958738 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.959706 (+ 968us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:10.959810 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.959834 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:01:10.959910 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:01:10.959978 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":21,"thread_start_us":109,"threads_started":1}
I20260810 18:01:10.960377 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:10.953044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.28.83.65:39931)
0810 18:01:10.953554 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:10.953561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:10.953617 (+ 56us) server_negotiation.cc:407] Connection header received
0810 18:01:10.953738 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:10.953744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:10.953953 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:10.954115 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:10.954527 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.956166 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.957447 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:10.958208 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:10.958352 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:10.958493 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:10.960152 (+ 1659us) server_negotiation.cc:299] Negotiation successful
0810 18:01:10.960255 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":96,"threads_started":1}
I20260810 18:01:10.962260 30443 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa45521ab5c45eb8682c41684dd574d (DEFAULT_TABLE table=test-table [id=f8d30a26ce9d4eb287c4d06a09082fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:10.962677 30297 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa45521ab5c45eb8682c41684dd574d (DEFAULT_TABLE table=test-table [id=f8d30a26ce9d4eb287c4d06a09082fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:10.963477 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa45521ab5c45eb8682c41684dd574d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:10.962997 30588 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa45521ab5c45eb8682c41684dd574d (DEFAULT_TABLE table=test-table [id=f8d30a26ce9d4eb287c4d06a09082fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:10.963827 30297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa45521ab5c45eb8682c41684dd574d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:10.964215 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa45521ab5c45eb8682c41684dd574d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:10.969834 30683 tablet_bootstrap.cc:492] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: Bootstrap starting.
I20260810 18:01:10.972218 30683 tablet_bootstrap.cc:654] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:10.972568 30684 tablet_bootstrap.cc:492] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: Bootstrap starting.
I20260810 18:01:10.973183 30683 log.cc:824] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:10.973755 30685 tablet_bootstrap.cc:492] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: Bootstrap starting.
I20260810 18:01:10.974735 30684 tablet_bootstrap.cc:654] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:10.974951 30683 tablet_bootstrap.cc:492] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: No bootstrap required, opened a new log
I20260810 18:01:10.975157 30683 ts_tablet_manager.cc:1397] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:01:10.975589 30684 log.cc:824] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:10.975858 30685 tablet_bootstrap.cc:654] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:10.976647 30685 log.cc:824] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:10.977758 30684 tablet_bootstrap.cc:492] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: No bootstrap required, opened a new log
I20260810 18:01:10.977946 30684 ts_tablet_manager.cc:1397] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:01:10.978488 30685 tablet_bootstrap.cc:492] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: No bootstrap required, opened a new log
I20260810 18:01:10.978686 30685 ts_tablet_manager.cc:1397] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:01:10.979718 30683 raft_consensus.cc:348] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } }
I20260810 18:01:10.980007 30683 raft_consensus.cc:374] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:10.980064 30683 raft_consensus.cc:729] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bc2c83960241fa8868978f3b22a011, State: Initialized, Role: FOLLOWER
I20260810 18:01:10.980504 30683 consensus_queue.cc:260] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } }
I20260810 18:01:10.981196 30366 heartbeater.cc:503] Master 127.28.83.126:42251 was elected leader, sending a full tablet report...
I20260810 18:01:10.981532 30683 ts_tablet_manager.cc:1428] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 18:01:10.981822 30684 raft_consensus.cc:348] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } }
I20260810 18:01:10.982085 30684 raft_consensus.cc:374] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:10.982156 30684 raft_consensus.cc:729] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a407492eff84a138b1a88deff8a2dbe, State: Initialized, Role: FOLLOWER
I20260810 18:01:10.982187 30685 raft_consensus.cc:348] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } }
I20260810 18:01:10.982427 30685 raft_consensus.cc:374] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:10.982486 30685 raft_consensus.cc:729] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c89f94e1d7c4b53b651eae6b870a708, State: Initialized, Role: FOLLOWER
I20260810 18:01:10.982611 30684 consensus_queue.cc:260] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } }
I20260810 18:01:10.982888 30685 consensus_queue.cc:260] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } }
I20260810 18:01:10.983307 30659 heartbeater.cc:503] Master 127.28.83.126:42251 was elected leader, sending a full tablet report...
I20260810 18:01:10.983709 30684 ts_tablet_manager.cc:1428] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:01:10.983796 30685 ts_tablet_manager.cc:1428] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260810 18:01:10.984161 30510 heartbeater.cc:503] Master 127.28.83.126:42251 was elected leader, sending a full tablet report...
I20260810 18:01:11.108043 30690 raft_consensus.cc:482] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:11.108269 30690 raft_consensus.cc:504] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } }
I20260810 18:01:11.109383 30690 leader_election.cc:288] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bc2c83960241fa8868978f3b22a011 (127.28.83.65:39931), 0c89f94e1d7c4b53b651eae6b870a708 (127.28.83.66:41157)
I20260810 18:01:11.113626 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.109844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:45003 (local address 127.28.83.66:41157)
0810 18:01:11.109981 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.109987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.110343 (+ 356us) server_negotiation.cc:407] Connection header received
0810 18:01:11.110412 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.110414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.110462 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.110546 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:11.110843 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.111391 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.112433 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.113032 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.113163 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.113204 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.113460 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.113526 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:01:11.113651 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.109733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41157 (local address 127.28.83.67:45003)
0810 18:01:11.110206 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:11.110220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:11.110336 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:11.110562 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:11.110564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:11.110594 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:11.110731 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.110738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.111504 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.111506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.112303 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.112311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.113160 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.113164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.113289 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.113316 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:01:11.113398 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:01:11.113486 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":33,"thread_start_us":134,"threads_started":1}
I20260810 18:01:11.114359 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.109733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:39931 (local address 127.28.83.67:46149)
0810 18:01:11.109962 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:11.109978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:11.110077 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:11.110368 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:11.110371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:11.110391 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:11.110523 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.110533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.111348 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.111352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.112955 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.112966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.113963 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.113969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.114070 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.114085 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 18:01:11.114171 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 18:01:11.114237 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260810 18:01:11.114313 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa45521ab5c45eb8682c41684dd574d" candidate_uuid: "0a407492eff84a138b1a88deff8a2dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c89f94e1d7c4b53b651eae6b870a708" is_pre_election: true
I20260810 18:01:11.114408 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.109937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:46149 (local address 127.28.83.65:39931)
0810 18:01:11.110054 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.110058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.110073 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:01:11.110177 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.110180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.110253 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.110340 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:11.110644 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.111238 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.113102 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.113762 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.113891 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.113951 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.114190 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.114290 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:01:11.114651 30459 raft_consensus.cc:2454] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a407492eff84a138b1a88deff8a2dbe in term 0.
I20260810 18:01:11.115037 30317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa45521ab5c45eb8682c41684dd574d" candidate_uuid: "0a407492eff84a138b1a88deff8a2dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bc2c83960241fa8868978f3b22a011" is_pre_election: true
I20260810 18:01:11.115175 30538 leader_election.cc:302] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a407492eff84a138b1a88deff8a2dbe, 0c89f94e1d7c4b53b651eae6b870a708; no voters:
I20260810 18:01:11.115321 30317 raft_consensus.cc:2454] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a407492eff84a138b1a88deff8a2dbe in term 0.
I20260810 18:01:11.115490 30690 raft_consensus.cc:2781] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:11.115607 30690 raft_consensus.cc:482] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:11.115660 30690 raft_consensus.cc:3037] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:11.116806 30690 raft_consensus.cc:504] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } }
I20260810 18:01:11.117136 30690 leader_election.cc:288] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [CANDIDATE]: Term 1 election: Requested vote from peers 49bc2c83960241fa8868978f3b22a011 (127.28.83.65:39931), 0c89f94e1d7c4b53b651eae6b870a708 (127.28.83.66:41157)
I20260810 18:01:11.117507 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa45521ab5c45eb8682c41684dd574d" candidate_uuid: "0a407492eff84a138b1a88deff8a2dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c89f94e1d7c4b53b651eae6b870a708"
I20260810 18:01:11.117573 30317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa45521ab5c45eb8682c41684dd574d" candidate_uuid: "0a407492eff84a138b1a88deff8a2dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bc2c83960241fa8868978f3b22a011"
I20260810 18:01:11.117645 30459 raft_consensus.cc:3037] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:11.117715 30317 raft_consensus.cc:3037] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:11.118705 30317 raft_consensus.cc:2454] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a407492eff84a138b1a88deff8a2dbe in term 1.
I20260810 18:01:11.119040 30537 leader_election.cc:302] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a407492eff84a138b1a88deff8a2dbe, 49bc2c83960241fa8868978f3b22a011; no voters:
I20260810 18:01:11.119230 30690 raft_consensus.cc:2781] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:11.119267 30459 raft_consensus.cc:2454] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a407492eff84a138b1a88deff8a2dbe in term 1.
I20260810 18:01:11.119486 30690 raft_consensus.cc:686] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 1 LEADER]: Becoming Leader. State: Replica: 0a407492eff84a138b1a88deff8a2dbe, State: Running, Role: LEADER
I20260810 18:01:11.119865 30690 consensus_queue.cc:237] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } }
I20260810 18:01:11.122978 30152 catalog_manager.cc:5495] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a407492eff84a138b1a88deff8a2dbe (127.28.83.67). New cstate: current_term: 1 leader_uuid: "0a407492eff84a138b1a88deff8a2dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } health_report { overall_health: UNKNOWN } } }
W20260810 18:01:11.144125 30660 tablet.cc:2369] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:11.157126 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.153362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.28.83.67:33727)
0810 18:01:11.153500 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.153503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.153628 (+ 125us) server_negotiation.cc:407] Connection header received
0810 18:01:11.153790 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.153793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.153851 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.153940 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:11.154170 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.154663 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.155715 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.155944 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.156018 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.156321 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:11.156786 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:11.156874 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.156955 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.157005 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 18:01:11.162835 30511 tablet.cc:2369] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:11.163334 30317 raft_consensus.cc:1261] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Refusing update from remote peer 0a407492eff84a138b1a88deff8a2dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:11.163998 30690 consensus_queue.cc:1035] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:11.164798 30459 raft_consensus.cc:1261] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 FOLLOWER]: Refusing update from remote peer 0a407492eff84a138b1a88deff8a2dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:11.165441 30690 consensus_queue.cc:1035] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:11.171911 30696 mvcc.cc:205] Tried to move back new op lower bound from 7317032432277082112 to 7317032432109342720. Current Snapshot: MvccSnapshot[applied={T|T < 7317032432277082112}]
I20260810 18:01:11.172981 30697 mvcc.cc:205] Tried to move back new op lower bound from 7317032432277082112 to 7317032432109342720. Current Snapshot: MvccSnapshot[applied={T|T < 7317032432277082112}]
I20260810 18:01:11.174721 30699 mvcc.cc:205] Tried to move back new op lower bound from 7317032432277082112 to 7317032432109342720. Current Snapshot: MvccSnapshot[applied={T|T < 7317032432277082112}]
I20260810 18:01:11.185896 30152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:11.186185 30152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:11.189088 30152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905: Sending DeleteTablet for 3 replicas of tablet 9aa45521ab5c45eb8682c41684dd574d
I20260810 18:01:11.190092 30297 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa45521ab5c45eb8682c41684dd574d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:11 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:55728
I20260810 18:01:11.190116 30443 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa45521ab5c45eb8682c41684dd574d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:11 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:34352
I20260810 18:01:11.190623 30705 tablet_replica.cc:330] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: stopping tablet replica
I20260810 18:01:11.190881 30706 tablet_replica.cc:330] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: stopping tablet replica
I20260810 18:01:11.190884 30588 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa45521ab5c45eb8682c41684dd574d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:11 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:55210
I20260810 18:01:11.191152 30705 raft_consensus.cc:2229] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:11.191151 30706 raft_consensus.cc:2229] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:11.191468 30705 raft_consensus.cc:2258] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:11.191468 30706 raft_consensus.cc:2258] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:11.191941 30152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:11.192373 30152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:11.192943 30706 ts_tablet_manager.cc:1905] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:11.193646 30705 ts_tablet_manager.cc:1905] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:11.196184 30706 ts_tablet_manager.cc:1918] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:11.196475 30705 ts_tablet_manager.cc:1918] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:11.196663 30705 log.cc:1196] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/9aa45521ab5c45eb8682c41684dd574d
I20260810 18:01:11.196993 30706 log.cc:1196] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/9aa45521ab5c45eb8682c41684dd574d
I20260810 18:01:11.197153 30705 ts_tablet_manager.cc:1939] T 9aa45521ab5c45eb8682c41684dd574d P 0c89f94e1d7c4b53b651eae6b870a708: Deleting consensus metadata
I20260810 18:01:11.197338 30706 ts_tablet_manager.cc:1939] T 9aa45521ab5c45eb8682c41684dd574d P 49bc2c83960241fa8868978f3b22a011: Deleting consensus metadata
I20260810 18:01:11.198128 30297 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7d3b49b464d7d8face3d8fe534be2 (DEFAULT_TABLE table=test-table [id=9bdf8ba449a549979d6d5171a84dcc98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:11.198380 30297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7d3b49b464d7d8face3d8fe534be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:11.198448 30138 catalog_manager.cc:4841] TS 49bc2c83960241fa8868978f3b22a011 (127.28.83.65:39931): tablet 9aa45521ab5c45eb8682c41684dd574d (table test-table [id=f8d30a26ce9d4eb287c4d06a09082fb2]) successfully deleted
I20260810 18:01:11.191879 30707 tablet_replica.cc:330] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: stopping tablet replica
I20260810 18:01:11.199034 30707 raft_consensus.cc:2229] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:11.199425 30707 raft_consensus.cc:2258] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:11.200027 30588 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7d3b49b464d7d8face3d8fe534be2 (DEFAULT_TABLE table=test-table [id=9bdf8ba449a549979d6d5171a84dcc98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:11.200314 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7d3b49b464d7d8face3d8fe534be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:11.200527 30683 tablet_bootstrap.cc:492] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: Bootstrap starting.
I20260810 18:01:11.201470 30683 tablet_bootstrap.cc:654] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:11.202492 30684 tablet_bootstrap.cc:492] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: Bootstrap starting.
I20260810 18:01:11.202680 30683 tablet_bootstrap.cc:492] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: No bootstrap required, opened a new log
I20260810 18:01:11.202687 30139 catalog_manager.cc:4841] TS 0c89f94e1d7c4b53b651eae6b870a708 (127.28.83.66:41157): tablet 9aa45521ab5c45eb8682c41684dd574d (table test-table [id=f8d30a26ce9d4eb287c4d06a09082fb2]) successfully deleted
I20260810 18:01:11.202765 30683 ts_tablet_manager.cc:1397] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:11.203161 30443 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7d3b49b464d7d8face3d8fe534be2 (DEFAULT_TABLE table=test-table [id=9bdf8ba449a549979d6d5171a84dcc98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:11.203402 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7d3b49b464d7d8face3d8fe534be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:11.203433 30684 tablet_bootstrap.cc:654] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:11.203493 30683 raft_consensus.cc:348] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.203596 30683 raft_consensus.cc:374] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:11.203626 30683 raft_consensus.cc:729] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bc2c83960241fa8868978f3b22a011, State: Initialized, Role: FOLLOWER
I20260810 18:01:11.203753 30683 consensus_queue.cc:260] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.204355 30683 ts_tablet_manager.cc:1428] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:11.205695 30685 tablet_bootstrap.cc:492] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: Bootstrap starting.
I20260810 18:01:11.206508 30684 tablet_bootstrap.cc:492] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: No bootstrap required, opened a new log
I20260810 18:01:11.206609 30684 ts_tablet_manager.cc:1397] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 18:01:11.206633 30685 tablet_bootstrap.cc:654] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:11.206830 30707 ts_tablet_manager.cc:1905] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:11.207046 30684 raft_consensus.cc:348] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.207158 30684 raft_consensus.cc:374] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:11.207197 30684 raft_consensus.cc:729] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a407492eff84a138b1a88deff8a2dbe, State: Initialized, Role: FOLLOWER
I20260810 18:01:11.207324 30684 consensus_queue.cc:260] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.207603 30684 ts_tablet_manager.cc:1428] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:11.208436 30685 tablet_bootstrap.cc:492] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: No bootstrap required, opened a new log
I20260810 18:01:11.208529 30685 ts_tablet_manager.cc:1397] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 18:01:11.209038 30685 raft_consensus.cc:348] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.209165 30685 raft_consensus.cc:374] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:11.209208 30685 raft_consensus.cc:729] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c89f94e1d7c4b53b651eae6b870a708, State: Initialized, Role: FOLLOWER
I20260810 18:01:11.209354 30685 consensus_queue.cc:260] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.209659 30685 ts_tablet_manager.cc:1428] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:11.210270 30707 ts_tablet_manager.cc:1918] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:11.210414 30707 log.cc:1196] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/9aa45521ab5c45eb8682c41684dd574d
I20260810 18:01:11.210799 30707 ts_tablet_manager.cc:1939] T 9aa45521ab5c45eb8682c41684dd574d P 0a407492eff84a138b1a88deff8a2dbe: Deleting consensus metadata
I20260810 18:01:11.211882 30138 catalog_manager.cc:4841] TS 0a407492eff84a138b1a88deff8a2dbe (127.28.83.67:33727): tablet 9aa45521ab5c45eb8682c41684dd574d (table test-table [id=f8d30a26ce9d4eb287c4d06a09082fb2]) successfully deleted
I20260810 18:01:11.370697 30691 raft_consensus.cc:482] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:11.370886 30691 raft_consensus.cc:504] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.372063 30691 leader_election.cc:288] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bc2c83960241fa8868978f3b22a011 (127.28.83.65:39931), 0a407492eff84a138b1a88deff8a2dbe (127.28.83.67:33727)
I20260810 18:01:11.376780 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.372922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:45089 (local address 127.28.83.67:33727)
0810 18:01:11.373081 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.373085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.373239 (+ 154us) server_negotiation.cc:407] Connection header received
0810 18:01:11.373320 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.373325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.373401 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.373553 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:11.373929 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.374437 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.375678 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.376071 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.376214 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.376300 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.376589 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.376661 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 18:01:11.376828 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.372792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:33727 (local address 127.28.83.66:45089)
0810 18:01:11.373074 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:11.373113 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:11.373215 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:11.373547 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:11.373549 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:11.373563 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:11.373803 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.373809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.374552 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.374556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.375541 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.375550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.376276 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.376280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.376450 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.376468 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:01:11.376559 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 18:01:11.376643 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260810 18:01:11.377024 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.372555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:35897 (local address 127.28.83.65:39931)
0810 18:01:11.372693 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.372697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.372753 (+ 56us) server_negotiation.cc:407] Connection header received
0810 18:01:11.373013 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.373016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.373070 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.373125 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:11.373885 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.374507 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.375961 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.376381 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.376512 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.376571 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.376851 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.376922 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:01:11.377096 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.372368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:39931 (local address 127.28.83.66:35897)
0810 18:01:11.372636 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:11.372651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:11.372842 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:11.373420 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:11.373426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:11.373470 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:11.373751 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.373760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.374642 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.374648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.375825 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.375835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.376507 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.376510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.376719 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.376737 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:01:11.376833 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 18:01:11.376899 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":113,"threads_started":1}
I20260810 18:01:11.377504 30317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7d3b49b464d7d8face3d8fe534be2" candidate_uuid: "0c89f94e1d7c4b53b651eae6b870a708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bc2c83960241fa8868978f3b22a011" is_pre_election: true
I20260810 18:01:11.377683 30317 raft_consensus.cc:2454] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c89f94e1d7c4b53b651eae6b870a708 in term 0.
I20260810 18:01:11.377636 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7d3b49b464d7d8face3d8fe534be2" candidate_uuid: "0c89f94e1d7c4b53b651eae6b870a708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a407492eff84a138b1a88deff8a2dbe" is_pre_election: true
I20260810 18:01:11.377892 30608 raft_consensus.cc:2454] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c89f94e1d7c4b53b651eae6b870a708 in term 0.
I20260810 18:01:11.378155 30392 leader_election.cc:302] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c89f94e1d7c4b53b651eae6b870a708, 49bc2c83960241fa8868978f3b22a011; no voters:
I20260810 18:01:11.378454 30691 raft_consensus.cc:2781] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:11.378510 30691 raft_consensus.cc:482] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:11.378537 30691 raft_consensus.cc:3037] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:11.379932 30691 raft_consensus.cc:504] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.380290 30691 leader_election.cc:288] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [CANDIDATE]: Term 1 election: Requested vote from peers 49bc2c83960241fa8868978f3b22a011 (127.28.83.65:39931), 0a407492eff84a138b1a88deff8a2dbe (127.28.83.67:33727)
I20260810 18:01:11.380687 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7d3b49b464d7d8face3d8fe534be2" candidate_uuid: "0c89f94e1d7c4b53b651eae6b870a708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a407492eff84a138b1a88deff8a2dbe"
I20260810 18:01:11.380686 30317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7d3b49b464d7d8face3d8fe534be2" candidate_uuid: "0c89f94e1d7c4b53b651eae6b870a708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bc2c83960241fa8868978f3b22a011"
I20260810 18:01:11.380813 30608 raft_consensus.cc:3037] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:11.380813 30317 raft_consensus.cc:3037] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:11.382236 30608 raft_consensus.cc:2454] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c89f94e1d7c4b53b651eae6b870a708 in term 1.
I20260810 18:01:11.382372 30317 raft_consensus.cc:2454] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c89f94e1d7c4b53b651eae6b870a708 in term 1.
I20260810 18:01:11.382555 30392 leader_election.cc:302] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a407492eff84a138b1a88deff8a2dbe, 0c89f94e1d7c4b53b651eae6b870a708; no voters:
I20260810 18:01:11.382738 30691 raft_consensus.cc:2781] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:11.382969 30691 raft_consensus.cc:686] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 LEADER]: Becoming Leader. State: Replica: 0c89f94e1d7c4b53b651eae6b870a708, State: Running, Role: LEADER
I20260810 18:01:11.383358 30691 consensus_queue.cc:237] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.385833 30152 catalog_manager.cc:5495] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c89f94e1d7c4b53b651eae6b870a708 (127.28.83.66). New cstate: current_term: 1 leader_uuid: "0c89f94e1d7c4b53b651eae6b870a708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } health_report { overall_health: UNKNOWN } } }
Aug 10 18:01:11 dist-test-slave-srgv krb5kdc[30106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:11.427591 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.419222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.28.83.126:42251)
0810 18:01:11.419343 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.419364 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.419696 (+ 332us) server_negotiation.cc:407] Connection header received
0810 18:01:11.419854 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.419857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.419903 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.419987 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:11.420708 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.421222 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.422053 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.422267 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.422374 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.424536 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:11.424540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:11.424543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:11.424571 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:11.425879 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:11.426334 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:11.426339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:11.426341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:11.426415 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:11.426696 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:11.426702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:11.426703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:11.426867 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:11.426991 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.427285 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.427448 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":43}
I20260810 18:01:11.427603 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.419139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42251 (local address 127.0.0.1:37040)
0810 18:01:11.419605 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:11.419619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:11.419759 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:11.420003 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:11.420006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:11.420486 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:11.420589 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.420597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.421334 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.421341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.421937 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.421945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.422376 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.422379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.422457 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.422979 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:11.422991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:11.424408 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:11.425983 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:11.425995 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:11.426001 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:11.426236 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:11.426519 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:11.426521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:11.426523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:11.426609 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:11.426989 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:11.426996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:11.427124 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 18:01:11.427240 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 18:01:11.427387 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":139,"threads_started":1}
I20260810 18:01:11.431841 30152 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:37040:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:01:11.432628 30152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:11.437595 30297 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca61378549b421095a62986c06f2f4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7aec9f23e1174a92a28343bcf3bc3a25]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:01:11.437697 30588 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca61378549b421095a62986c06f2f4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7aec9f23e1174a92a28343bcf3bc3a25]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:01:11.438011 30297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca61378549b421095a62986c06f2f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:11.438040 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca61378549b421095a62986c06f2f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:11.441299 30683 tablet_bootstrap.cc:492] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: Bootstrap starting.
I20260810 18:01:11.441591 30443 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca61378549b421095a62986c06f2f4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7aec9f23e1174a92a28343bcf3bc3a25]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:01:11.441886 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca61378549b421095a62986c06f2f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:11.442303 30683 tablet_bootstrap.cc:654] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: Neither blocks nor log segments found. Creating new log.
W20260810 18:01:11.442406 30367 tablet.cc:2369] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:11.443961 30684 tablet_bootstrap.cc:492] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: Bootstrap starting.
I20260810 18:01:11.445310 30684 tablet_bootstrap.cc:654] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:11.445457 30685 tablet_bootstrap.cc:492] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: Bootstrap starting.
I20260810 18:01:11.446741 30684 tablet_bootstrap.cc:492] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: No bootstrap required, opened a new log
I20260810 18:01:11.446854 30684 ts_tablet_manager.cc:1397] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:11.447021 30685 tablet_bootstrap.cc:654] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:11.447372 30684 raft_consensus.cc:348] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.447499 30684 raft_consensus.cc:374] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:11.447539 30684 raft_consensus.cc:729] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a407492eff84a138b1a88deff8a2dbe, State: Initialized, Role: FOLLOWER
I20260810 18:01:11.447670 30684 consensus_queue.cc:260] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.447973 30693 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.448107 30693 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:11.448341 30684 ts_tablet_manager.cc:1428] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:11.449446 30685 tablet_bootstrap.cc:492] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: No bootstrap required, opened a new log
I20260810 18:01:11.449559 30685 ts_tablet_manager.cc:1397] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 18:01:11.449684 30683 tablet_bootstrap.cc:492] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: No bootstrap required, opened a new log
I20260810 18:01:11.449774 30683 ts_tablet_manager.cc:1397] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260810 18:01:11.450369 30685 raft_consensus.cc:348] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.450482 30685 raft_consensus.cc:374] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:11.450444 30683 raft_consensus.cc:348] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.450517 30685 raft_consensus.cc:729] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c89f94e1d7c4b53b651eae6b870a708, State: Initialized, Role: FOLLOWER
I20260810 18:01:11.450546 30683 raft_consensus.cc:374] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:11.450575 30683 raft_consensus.cc:729] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bc2c83960241fa8868978f3b22a011, State: Initialized, Role: FOLLOWER
I20260810 18:01:11.450646 30685 consensus_queue.cc:260] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.450681 30683 consensus_queue.cc:260] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.450946 30685 ts_tablet_manager.cc:1428] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:11.451047 30683 ts_tablet_manager.cc:1428] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:01:11.451169 30691 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.451301 30691 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:11.451349 30689 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.451550 30689 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:11.528733 30693 raft_consensus.cc:482] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:11.528914 30693 raft_consensus.cc:504] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.529443 30693 leader_election.cc:288] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bc2c83960241fa8868978f3b22a011 (127.28.83.65:39931), 0c89f94e1d7c4b53b651eae6b870a708 (127.28.83.66:41157)
I20260810 18:01:11.529883 30317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca61378549b421095a62986c06f2f4c" candidate_uuid: "0a407492eff84a138b1a88deff8a2dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bc2c83960241fa8868978f3b22a011" is_pre_election: true
I20260810 18:01:11.529919 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca61378549b421095a62986c06f2f4c" candidate_uuid: "0a407492eff84a138b1a88deff8a2dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c89f94e1d7c4b53b651eae6b870a708" is_pre_election: true
I20260810 18:01:11.530043 30317 raft_consensus.cc:2454] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a407492eff84a138b1a88deff8a2dbe in term 0.
I20260810 18:01:11.530161 30459 raft_consensus.cc:2454] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a407492eff84a138b1a88deff8a2dbe in term 0.
I20260810 18:01:11.530354 30537 leader_election.cc:302] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a407492eff84a138b1a88deff8a2dbe, 49bc2c83960241fa8868978f3b22a011; no voters:
I20260810 18:01:11.530567 30693 raft_consensus.cc:2781] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:11.530629 30693 raft_consensus.cc:482] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:11.530658 30693 raft_consensus.cc:3037] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:11.532029 30693 raft_consensus.cc:504] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.532401 30693 leader_election.cc:288] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [CANDIDATE]: Term 1 election: Requested vote from peers 49bc2c83960241fa8868978f3b22a011 (127.28.83.65:39931), 0c89f94e1d7c4b53b651eae6b870a708 (127.28.83.66:41157)
I20260810 18:01:11.532806 30317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca61378549b421095a62986c06f2f4c" candidate_uuid: "0a407492eff84a138b1a88deff8a2dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bc2c83960241fa8868978f3b22a011"
I20260810 18:01:11.532894 30459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca61378549b421095a62986c06f2f4c" candidate_uuid: "0a407492eff84a138b1a88deff8a2dbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c89f94e1d7c4b53b651eae6b870a708"
I20260810 18:01:11.532974 30317 raft_consensus.cc:3037] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:11.533032 30459 raft_consensus.cc:3037] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:11.534155 30317 raft_consensus.cc:2454] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a407492eff84a138b1a88deff8a2dbe in term 1.
I20260810 18:01:11.534397 30459 raft_consensus.cc:2454] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a407492eff84a138b1a88deff8a2dbe in term 1.
I20260810 18:01:11.534518 30537 leader_election.cc:302] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a407492eff84a138b1a88deff8a2dbe, 49bc2c83960241fa8868978f3b22a011; no voters:
I20260810 18:01:11.534760 30693 raft_consensus.cc:2781] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:11.534873 30693 raft_consensus.cc:686] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [term 1 LEADER]: Becoming Leader. State: Replica: 0a407492eff84a138b1a88deff8a2dbe, State: Running, Role: LEADER
I20260810 18:01:11.535027 30693 consensus_queue.cc:237] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } }
I20260810 18:01:11.535732 30690 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: TxnStatusTablet state changed. Reason: New leader 0a407492eff84a138b1a88deff8a2dbe. Latest consensus state: current_term: 1 leader_uuid: "0a407492eff84a138b1a88deff8a2dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.535847 30690 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:01:11.536158 30726 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:01:11.536367 30726 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:01:11.536959 30152 catalog_manager.cc:5495] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a407492eff84a138b1a88deff8a2dbe (127.28.83.67). New cstate: current_term: 1 leader_uuid: "0a407492eff84a138b1a88deff8a2dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } health_report { overall_health: HEALTHY } } }
I20260810 18:01:11.556649 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.552929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.28.83.67:33727)
0810 18:01:11.553066 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.553069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.553086 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:01:11.553130 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.553132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.553174 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.553265 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:11.553583 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.554161 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.555101 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.555598 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.555745 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.556121 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:11.556205 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:11.556316 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.556468 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.556521 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:01:11.556684 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.552496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:33727 (local address 127.0.0.1:55240)
0810 18:01:11.552664 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:11.552678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:11.552771 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:11.553295 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:11.553298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:11.553306 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:11.553465 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.553473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.554270 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.554276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.554965 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.554974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.555759 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.555763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.555863 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.555911 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:11.556339 (+ 428us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:11.556361 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:01:11.556449 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 18:01:11.556536 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 18:01:11.559409 30317 raft_consensus.cc:1261] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Refusing update from remote peer 0a407492eff84a138b1a88deff8a2dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:11.559387 30459 raft_consensus.cc:1261] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 FOLLOWER]: Refusing update from remote peer 0a407492eff84a138b1a88deff8a2dbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:11.559826 30690 consensus_queue.cc:1035] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:11.560014 30693 consensus_queue.cc:1035] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:11.561766 30689 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: TxnStatusTablet state changed. Reason: New leader 0a407492eff84a138b1a88deff8a2dbe. Latest consensus state: current_term: 1 leader_uuid: "0a407492eff84a138b1a88deff8a2dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.561810 30691 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: TxnStatusTablet state changed. Reason: New leader 0a407492eff84a138b1a88deff8a2dbe. Latest consensus state: current_term: 1 leader_uuid: "0a407492eff84a138b1a88deff8a2dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.561890 30689 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:11.561908 30691 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:11.564052 30690 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a407492eff84a138b1a88deff8a2dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.564200 30690 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:01:11.564947 30689 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a407492eff84a138b1a88deff8a2dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.565062 30689 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 49bc2c83960241fa8868978f3b22a011: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:11.566594 30693 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a407492eff84a138b1a88deff8a2dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.566704 30693 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 0a407492eff84a138b1a88deff8a2dbe: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:01:11.567516 30691 tablet_replica.cc:439] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a407492eff84a138b1a88deff8a2dbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 } } peers { permanent_uuid: "0c89f94e1d7c4b53b651eae6b870a708" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41157 } } peers { permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 } } }
I20260810 18:01:11.567674 30691 tablet_replica.cc:442] T 1ca61378549b421095a62986c06f2f4c P 0c89f94e1d7c4b53b651eae6b870a708: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:11.574548 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.570978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.28.83.66:41157)
0810 18:01:11.571096 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.571114 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.571125 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:01:11.571191 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.571193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.571241 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.571356 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:11.571605 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.572124 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.572951 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.573202 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.573313 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.573634 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:11.574078 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:11.574219 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.574279 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.574332 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:01:11.587251 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.582801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:46595 (local address 127.28.83.67:33727)
0810 18:01:11.582951 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.582956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.583353 (+ 397us) server_negotiation.cc:407] Connection header received
0810 18:01:11.583476 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.583478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.583576 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.583656 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:11.584087 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.584629 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.586145 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.586360 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.586447 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.586849 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:11.586918 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:11.586973 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.587092 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.587139 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:01:11.587847 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.582681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:33727 (local address 127.28.83.66:46595)
0810 18:01:11.583265 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:11.583281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:11.583391 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:11.583686 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:11.583689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:11.583696 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:11.583973 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.583981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.584747 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.584751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.586005 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.586017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.586476 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.586480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.586651 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.586706 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:11.587000 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:11.587006 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:01:11.587072 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:01:11.587152 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":187,"threads_started":1}
I20260810 18:01:11.595129 30317 raft_consensus.cc:1261] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Refusing update from remote peer 0c89f94e1d7c4b53b651eae6b870a708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:11.595409 30608 raft_consensus.cc:1261] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [term 1 FOLLOWER]: Refusing update from remote peer 0c89f94e1d7c4b53b651eae6b870a708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:11.595867 30691 consensus_queue.cc:1035] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bc2c83960241fa8868978f3b22a011" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:11.596143 30713 consensus_queue.cc:1035] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a407492eff84a138b1a88deff8a2dbe" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 33727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:11.620183 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.616252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:46195 (local address 127.28.83.66:41157)
0810 18:01:11.616399 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.616404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.616647 (+ 243us) server_negotiation.cc:407] Connection header received
0810 18:01:11.616766 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.616769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.616808 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.616864 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:11.617166 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.617864 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.619092 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.619330 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.619446 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.619738 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:11.619818 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:11.619897 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.619990 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.620039 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:01:11.620187 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.616191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41157 (local address 127.28.83.67:46195)
0810 18:01:11.616551 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:11.616567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:11.616674 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:11.616892 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:11.616895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:11.616903 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:11.617009 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.617015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.617994 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.617998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.618972 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:11.618981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.619446 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.619450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:11.619555 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.619598 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:11.619907 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:11.619916 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:01:11.619980 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 18:01:11.620062 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":113,"threads_started":1}
I20260810 18:01:11.650492 30149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:11.650691 30149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:11.653496 30149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fba2bcf92e364250bfd9402a1af5d905: Sending DeleteTablet for 3 replicas of tablet 2af7d3b49b464d7d8face3d8fe534be2
I20260810 18:01:11.654289 30297 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af7d3b49b464d7d8face3d8fe534be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:11 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:55728
I20260810 18:01:11.654562 30706 tablet_replica.cc:330] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: stopping tablet replica
I20260810 18:01:11.654785 30706 raft_consensus.cc:2229] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:11.655026 30706 raft_consensus.cc:2258] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:11.655433 30443 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af7d3b49b464d7d8face3d8fe534be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:11 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:34352
I20260810 18:01:11.655715 30705 tablet_replica.cc:330] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: stopping tablet replica
I20260810 18:01:11.655860 30705 raft_consensus.cc:2229] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:11.656226 30705 raft_consensus.cc:2258] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:11.657581 30705 ts_tablet_manager.cc:1905] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:11.661724 30588 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af7d3b49b464d7d8face3d8fe534be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:11 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:55210
I20260810 18:01:11.661933 30707 tablet_replica.cc:330] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: stopping tablet replica
I20260810 18:01:11.662068 30707 raft_consensus.cc:2229] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:11.662230 30707 raft_consensus.cc:2258] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:11.663004 30706 ts_tablet_manager.cc:1905] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:11.663055 30707 ts_tablet_manager.cc:1905] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:11.666078 30707 ts_tablet_manager.cc:1918] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:11.666172 30707 log.cc:1196] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/2af7d3b49b464d7d8face3d8fe534be2
I20260810 18:01:11.666429 30707 ts_tablet_manager.cc:1939] T 2af7d3b49b464d7d8face3d8fe534be2 P 0a407492eff84a138b1a88deff8a2dbe: Deleting consensus metadata
I20260810 18:01:11.667255 30138 catalog_manager.cc:4841] TS 0a407492eff84a138b1a88deff8a2dbe (127.28.83.67:33727): tablet 2af7d3b49b464d7d8face3d8fe534be2 (table test-table [id=9bdf8ba449a549979d6d5171a84dcc98]) successfully deleted
I20260810 18:01:11.666085 30706 ts_tablet_manager.cc:1918] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:11.667703 30706 log.cc:1196] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/2af7d3b49b464d7d8face3d8fe534be2
I20260810 18:01:11.668030 30706 ts_tablet_manager.cc:1939] T 2af7d3b49b464d7d8face3d8fe534be2 P 49bc2c83960241fa8868978f3b22a011: Deleting consensus metadata
Aug 10 18:01:11 dist-test-slave-srgv krb5kdc[30106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:11.668686 30138 catalog_manager.cc:4841] TS 49bc2c83960241fa8868978f3b22a011 (127.28.83.65:39931): tablet 2af7d3b49b464d7d8face3d8fe534be2 (table test-table [id=9bdf8ba449a549979d6d5171a84dcc98]) successfully deleted
I20260810 18:01:11.674894 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.658324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.28.83.65:39931)
0810 18:01:11.658437 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.658442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.659163 (+ 721us) server_negotiation.cc:407] Connection header received
0810 18:01:11.659356 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.659359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.659420 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.659473 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:11.660759 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.661556 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.663524 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.663858 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.663929 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.669558 (+ 5629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:11.669571 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:11.669588 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:11.669617 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:11.672286 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:11.672860 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:11.672866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:11.672868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:11.673108 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:11.673458 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:11.673462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:11.673463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:11.673795 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:11.673992 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.674416 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.674558 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 18:01:11.676429 30360 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55738
I20260810 18:01:11.683091 30705 ts_tablet_manager.cc:1918] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:11.683200 30705 log.cc:1196] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/2af7d3b49b464d7d8face3d8fe534be2
I20260810 18:01:11.683485 30705 ts_tablet_manager.cc:1939] T 2af7d3b49b464d7d8face3d8fe534be2 P 0c89f94e1d7c4b53b651eae6b870a708: Deleting consensus metadata
I20260810 18:01:11.684386 30139 catalog_manager.cc:4841] TS 0c89f94e1d7c4b53b651eae6b870a708 (127.28.83.66:41157): tablet 2af7d3b49b464d7d8face3d8fe534be2 (table test-table [id=9bdf8ba449a549979d6d5171a84dcc98]) successfully deleted
I20260810 18:01:11.694139 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:11.681968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.28.83.126:42251)
0810 18:01:11.682097 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:11.682102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:11.683684 (+ 1582us) server_negotiation.cc:407] Connection header received
0810 18:01:11.683914 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:11.683917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:11.683971 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:11.684027 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:11.685726 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.686506 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.687733 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:11.688054 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:11.688138 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:11.690178 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:11.690182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:11.690185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:11.690214 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:11.692164 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:11.692700 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:11.692704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:11.692706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:11.692794 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:11.693103 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:11.693106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:11.693108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:11.693330 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:11.693470 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:11.693843 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 18:01:11.693967 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 18:01:11.695271 30149 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37042
I20260810 18:01:11.697682 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 30227
I20260810 18:01:11.708570 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 30375
I20260810 18:01:11.716012 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 30519
I20260810 18:01:11.723794 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 30119
2026-08-10T18:01:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2998 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:11 dist-test-slave-srgv krb5kdc[30749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:11 dist-test-slave-srgv krb5kdc[30749](info): set up 2 sockets
Aug 10 18:01:11 dist-test-slave-srgv krb5kdc[30749](info): commencing operation
krb5kdc: starting...
W20260810 18:01:12.773690 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:12 dist-test-slave-srgv krb5kdc[30749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:12.900569 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43601
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:43601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:13.012833 30762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:13.013101 30762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:13.013142 30762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:13.017017 30762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:13.017076 30762 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:01:13.017100 30762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:13.017118 30762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:13.017135 30762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:13.017150 30762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:13.022336 30762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:43601
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:13.023309 30762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:13.024277 30762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:13.032740 30768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:13.032754 30767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:13.033070 30762 server_base.cc:1034] running on GCE node
W20260810 18:01:13.032759 30770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:13.033715 30762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:13.034835 30762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:13.036037 30762 hybrid_clock.cc:648] HybridClock initialized: now 1786384873036010 us; error 45 us; skew 500 ppm
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:13.042136 30762 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:13.043210 30762 webserver.cc:460] Webserver started at http://127.28.83.126:45325/ using document root <none> and password file <none>
I20260810 18:01:13.043792 30762 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:13.043886 30762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:13.044134 30762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:13.046334 30762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "698c0532b4ce4a26ba80fb38a08be989"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "c31a59f6a44040d492fde38faa06e895"
server_key_iv: "b806631f844d01b38b0b4078dd06234f"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.046907 30762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "698c0532b4ce4a26ba80fb38a08be989"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "c31a59f6a44040d492fde38faa06e895"
server_key_iv: "b806631f844d01b38b0b4078dd06234f"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.050701 30762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:01:13.053090 30778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:13.054387 30762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:13.054534 30762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "698c0532b4ce4a26ba80fb38a08be989"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "c31a59f6a44040d492fde38faa06e895"
server_key_iv: "b806631f844d01b38b0b4078dd06234f"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.054658 30762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:13.070650 30762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:13.073843 30762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:13.074101 30762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:13.082734 30762 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:43601
I20260810 18:01:13.082768 30841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:43601 every 8 connection(s)
I20260810 18:01:13.083726 30762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:13.086587 30842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:13.087311 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 30762
I20260810 18:01:13.087433 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:13.087740 29005 external_mini_cluster.cc:1428] Setting key e93073dc8e6a6afeb8d7c9a5802cc2bf
I20260810 18:01:13.092974 30842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989: Bootstrap starting.
I20260810 18:01:13.095491 30842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:13.096259 30842 log.cc:824] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:13.098526 30842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989: No bootstrap required, opened a new log
I20260810 18:01:13.102001 30842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698c0532b4ce4a26ba80fb38a08be989" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43601 } }
I20260810 18:01:13.102321 30842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:13.102385 30842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698c0532b4ce4a26ba80fb38a08be989, State: Initialized, Role: FOLLOWER
I20260810 18:01:13.102893 30842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698c0532b4ce4a26ba80fb38a08be989" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43601 } }
I20260810 18:01:13.103024 30842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:13.103106 30842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:13.103204 30842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:13.103791 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.089336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.28.83.126:43601)
0810 18:01:13.089849 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.089865 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.089902 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:01:13.090539 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.090555 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.090907 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.091258 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:13.092071 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.093082 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.094047 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.094527 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.094711 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.097651 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:13.097656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:13.097670 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:13.097695 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:13.100458 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.101012 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.101017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.101022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.101135 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.101497 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.101499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.101501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.101923 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:13.102077 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.102487 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.102831 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":148,"threads_started":1}
I20260810 18:01:13.104262 30842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698c0532b4ce4a26ba80fb38a08be989" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43601 } }
I20260810 18:01:13.104614 30842 leader_election.cc:302] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 698c0532b4ce4a26ba80fb38a08be989; no voters:
I20260810 18:01:13.104929 30842 leader_election.cc:288] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:13.105026 30847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:13.105255 30847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [term 1 LEADER]: Becoming Leader. State: Replica: 698c0532b4ce4a26ba80fb38a08be989, State: Running, Role: LEADER
I20260810 18:01:13.105587 30847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698c0532b4ce4a26ba80fb38a08be989" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43601 } }
I20260810 18:01:13.106295 30842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:13.106913 30850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 698c0532b4ce4a26ba80fb38a08be989. Latest consensus state: current_term: 1 leader_uuid: "698c0532b4ce4a26ba80fb38a08be989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698c0532b4ce4a26ba80fb38a08be989" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43601 } } }
I20260810 18:01:13.107045 30850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:13.107343 30851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "698c0532b4ce4a26ba80fb38a08be989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698c0532b4ce4a26ba80fb38a08be989" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43601 } } }
I20260810 18:01:13.107460 30851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:13.108739 30855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:13.111843 30855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:13.117091 30855 catalog_manager.cc:1347] Generated new cluster ID: 7a42fd1e1a5f44528bb8e24bd8c1382b
I20260810 18:01:13.117161 30855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:13.130513 30855 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:13.131822 30855 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:13.150791 30855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989: Generated new TSK 0
I20260810 18:01:13.151582 30855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:13.184782 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43601
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:13.297983 30871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:13.298245 30871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:13.298331 30871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:13.302336 30871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:13.302384 30871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:13.302459 30871 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:01:13.302500 30871 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:01:13.302570 30871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:13.308390 30871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.83.126:43601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:13.309520 30871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:13.310529 30871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:13.320016 30876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:13.320030 30879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:13.320309 30871 server_base.cc:1034] running on GCE node
W20260810 18:01:13.320037 30877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:13.320785 30871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:13.321381 30871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:13.322588 30871 hybrid_clock.cc:648] HybridClock initialized: now 1786384873322559 us; error 42 us; skew 500 ppm
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:13.325340 30871 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:13.326239 30871 webserver.cc:460] Webserver started at http://127.28.83.65:34777/ using document root <none> and password file <none>
I20260810 18:01:13.326787 30871 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:13.326885 30871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:13.327114 30871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:13.329099 30871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "e9ded6ba19b1a17a60cf12289a57ae04"
server_key_iv: "375d19469ec04d76e873d932b7af4136"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.329710 30871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "e9ded6ba19b1a17a60cf12289a57ae04"
server_key_iv: "375d19469ec04d76e873d932b7af4136"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.333524 30871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:01:13.335861 30886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:13.337045 30871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:13.337195 30871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "e9ded6ba19b1a17a60cf12289a57ae04"
server_key_iv: "375d19469ec04d76e873d932b7af4136"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.337317 30871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:13.352846 30871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:13.356284 30871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:13.356506 30871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:13.358099 30871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:13.358170 30871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:13.358266 30871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:13.358318 30871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:13.369567 30871 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:41275
I20260810 18:01:13.369642 31004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:41275 every 8 connection(s)
I20260810 18:01:13.370761 30871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:13.371724 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 30871
I20260810 18:01:13.371843 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:13.372186 29005 external_mini_cluster.cc:1428] Setting key c3f4fc90339b8b504ae53802b07d842e
I20260810 18:01:13.373239 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.358866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:56283 (local address 127.28.83.126:43601)
0810 18:01:13.359058 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.359064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.360389 (+ 1325us) server_negotiation.cc:407] Connection header received
0810 18:01:13.361848 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.361853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.361911 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.361997 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:13.363692 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.364332 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.365763 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.366099 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.366215 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.368838 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:13.368841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:13.368843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:13.368868 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:13.370699 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.371264 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.371271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.371276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.371395 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.372154 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.372157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.372158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.372411 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:13.372494 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.373002 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.373112 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:01:13.374207 30895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.359716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43601 (local address 127.28.83.65:56283)
0810 18:01:13.360206 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:13.360265 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:13.361575 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:13.362209 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:13.362219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:13.362758 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:13.363507 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.363529 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.364474 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.364478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.365636 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.365646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.366259 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.366264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.366439 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.367052 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:13.367069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:13.368632 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:13.370850 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.370857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.370873 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.371147 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.371842 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.371846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.371849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.372024 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.372523 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:13.372532 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:13.372777 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 18:01:13.372977 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 18:01:13.373307 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260810 18:01:13.375427 31005 heartbeater.cc:347] Connected to a master server at 127.28.83.126:43601
I20260810 18:01:13.375645 31005 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:13.376168 31005 heartbeater.cc:511] Master 127.28.83.126:43601 requested a full tablet report, sending...
I20260810 18:01:13.377837 30794 ts_manager.cc:194] Registered new tserver with Master: 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275)
I20260810 18:01:13.379107 30794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:56283
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:13.387708 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.378073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43601 (local address 127.28.83.65:58627)
0810 18:01:13.378365 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:13.378381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:13.378464 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:13.380135 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:13.380138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:13.380403 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:13.380549 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.380559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.381647 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.381652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.382679 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.382693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.383177 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.383180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.383256 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.383770 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:13.383784 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:13.384118 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:13.386137 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.386142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.386144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.386345 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.386634 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.386638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.386640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.386728 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.387243 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:13.387245 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:13.387332 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 18:01:13.387430 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 18:01:13.387547 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260810 18:01:13.387845 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.379647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:58627 (local address 127.28.83.126:43601)
0810 18:01:13.379798 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.379805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.379823 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:01:13.379868 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.379873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.379929 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.380012 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:13.380694 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.381505 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.382827 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.383044 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.383162 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.384240 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:13.384243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:13.384245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:13.384263 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:13.386019 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.386446 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.386450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.386452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.386532 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.386819 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.386822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.386824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.387093 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:13.387213 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.387511 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.387648 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:13.402292 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43601
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:13.518428 31015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:13.518692 31015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:13.518774 31015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:13.522707 31015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:13.522794 31015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:13.522869 31015 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:01:13.522912 31015 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:01:13.522980 31015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:13.528297 31015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.83.126:43601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:13.529364 31015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:13.530400 31015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:13.540151 31020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:13.540151 31023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:13.540205 31015 server_base.cc:1034] running on GCE node
W20260810 18:01:13.540145 31021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:13.540735 31015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:13.541563 31015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:13.542742 31015 hybrid_clock.cc:648] HybridClock initialized: now 1786384873542727 us; error 27 us; skew 500 ppm
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:13.545586 31015 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:13.546566 31015 webserver.cc:460] Webserver started at http://127.28.83.66:38727/ using document root <none> and password file <none>
I20260810 18:01:13.547158 31015 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:13.547225 31015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:13.547467 31015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:13.550017 31015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "d26a8527c619486ca2f15ef8189e210a"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "3a915a975b6f505d2aad9a802c4f36e5"
server_key_iv: "5a030bf526a091f12399c1961e1fbe16"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.550608 31015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "d26a8527c619486ca2f15ef8189e210a"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "3a915a975b6f505d2aad9a802c4f36e5"
server_key_iv: "5a030bf526a091f12399c1961e1fbe16"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.554396 31015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:01:13.556802 31031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:13.558089 31015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:13.558207 31015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "d26a8527c619486ca2f15ef8189e210a"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "3a915a975b6f505d2aad9a802c4f36e5"
server_key_iv: "5a030bf526a091f12399c1961e1fbe16"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.558324 31015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:13.573770 31015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:13.577001 31015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:13.577224 31015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:13.578764 31015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:13.578840 31015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:13.578907 31015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:13.578949 31015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:13.590358 31015 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:34447
I20260810 18:01:13.590469 31152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:34447 every 8 connection(s)
I20260810 18:01:13.591277 31015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:13.593084 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.579859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:49667 (local address 127.28.83.126:43601)
0810 18:01:13.580040 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.580045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.581008 (+ 963us) server_negotiation.cc:407] Connection header received
0810 18:01:13.582379 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.582382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.582446 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.582581 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:13.583826 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.584378 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.585635 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.585943 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.586065 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.589213 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:13.589216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:13.589218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:13.589243 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:13.590942 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.591546 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.591550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.591554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.591640 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.592081 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.592084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.592086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.592331 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:13.592427 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.592860 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.592966 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 18:01:13.593945 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.580305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43601 (local address 127.28.83.66:49667)
0810 18:01:13.580812 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:13.580873 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:13.582117 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:13.582755 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:13.582766 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:13.583118 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:13.583682 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.583696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.584510 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.584519 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.585510 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.585520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.586123 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.586127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.586329 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.586966 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:13.586982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:13.589020 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:13.591064 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.591073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.591087 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.591442 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.591737 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.591742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.591743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.591928 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.592424 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:13.592430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:13.592655 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 18:01:13.592810 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 18:01:13.593049 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260810 18:01:13.595191 31153 heartbeater.cc:347] Connected to a master server at 127.28.83.126:43601
I20260810 18:01:13.595351 31153 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:13.595840 31153 heartbeater.cc:511] Master 127.28.83.126:43601 requested a full tablet report, sending...
I20260810 18:01:13.596810 30795 ts_manager.cc:194] Registered new tserver with Master: d26a8527c619486ca2f15ef8189e210a (127.28.83.66:34447)
I20260810 18:01:13.597539 30795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:49667
I20260810 18:01:13.599354 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 31015
I20260810 18:01:13.599488 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:13.599768 29005 external_mini_cluster.cc:1428] Setting key 10bb70bd71457a770087b0aa06651ccf
I20260810 18:01:13.604848 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.597338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43601 (local address 127.28.83.66:33003)
0810 18:01:13.597668 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:13.597683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:13.597779 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:13.598023 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:13.598026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:13.598229 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:13.598325 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.598330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.599077 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.599081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.599761 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.599770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.600254 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.600257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.600375 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.600933 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:13.600947 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:13.601357 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:13.603301 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.603307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.603310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.603574 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.603860 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.603866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.603870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.603983 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.604443 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:13.604445 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:13.604523 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 18:01:13.604610 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 18:01:13.604717 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":169,"threads_started":1}
I20260810 18:01:13.605029 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.597375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:33003 (local address 127.28.83.126:43601)
0810 18:01:13.597509 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.597513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.597756 (+ 243us) server_negotiation.cc:407] Connection header received
0810 18:01:13.597875 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.597878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.597925 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.598007 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:13.598444 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.598949 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.599879 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.600137 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.600245 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.601511 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:13.601513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:13.601515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:13.601536 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:13.603166 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.603689 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.603692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.603694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.603767 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.604095 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.604098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.604099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.604341 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:13.604443 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.604697 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.604827 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:01:13.629014 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43601
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:13.744215 31162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:13.744488 31162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:13.744537 31162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:13.748482 31162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:13.748586 31162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:13.748646 31162 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:01:13.748674 31162 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:01:13.748740 31162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:13.754432 31162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.83.126:43601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:13.755404 31162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:13.756348 31162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:13.766320 31168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:13.766327 31167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:13.766335 31170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:13.766610 31162 server_base.cc:1034] running on GCE node
I20260810 18:01:13.766996 31162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:13.767628 31162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:13.768785 31162 hybrid_clock.cc:648] HybridClock initialized: now 1786384873768755 us; error 43 us; skew 500 ppm
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:13.771679 31162 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:13.772574 31162 webserver.cc:460] Webserver started at http://127.28.83.67:43313/ using document root <none> and password file <none>
I20260810 18:01:13.773128 31162 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:13.773190 31162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:13.773356 31162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:13.775688 31162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "cf2dc2e3ccb046308e78478bba960790"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "bcc3f0301ec742ec553f1bfcd918bbda"
server_key_iv: "d6570d7710d7c2f17796845860eda8c5"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.776255 31162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "cf2dc2e3ccb046308e78478bba960790"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "bcc3f0301ec742ec553f1bfcd918bbda"
server_key_iv: "d6570d7710d7c2f17796845860eda8c5"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.780210 31162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 18:01:13.782721 31177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:13.784006 31162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:13.784147 31162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "cf2dc2e3ccb046308e78478bba960790"
format_stamp: "Formatted at 2026-08-10 18:01:13 on dist-test-slave-srgv"
server_key: "bcc3f0301ec742ec553f1bfcd918bbda"
server_key_iv: "d6570d7710d7c2f17796845860eda8c5"
server_key_version: "encryptionkey@0"
I20260810 18:01:13.784272 31162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:13.800046 31162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:13.803596 31162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:13.803817 31162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:13.805302 31162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:13.805532 31162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:13.805660 31162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:13.805712 31162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:13.816596 31162 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:41503
I20260810 18:01:13.816679 31296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:41503 every 8 connection(s)
I20260810 18:01:13.817691 31162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:13.819988 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.806293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:36171 (local address 127.28.83.126:43601)
0810 18:01:13.806461 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.806466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.807866 (+ 1400us) server_negotiation.cc:407] Connection header received
0810 18:01:13.809124 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.809131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.809203 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.809294 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:13.810666 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.811335 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.812643 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.812878 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.812991 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.816344 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:13.816348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:13.816352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:13.816386 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:13.817988 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.818554 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.818558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.818560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.818670 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.819021 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.819024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.819025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.819230 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:13.819321 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.819764 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.819869 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:01:13.820780 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.807202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43601 (local address 127.28.83.67:36171)
0810 18:01:13.807704 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:13.807757 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:13.808860 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:13.809527 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:13.809534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:13.810033 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:13.810518 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.810532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.811458 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.811462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.812502 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.812512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.813050 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.813054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.813234 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.814438 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:13.814459 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:13.816164 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:13.818118 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.818123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.818138 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.818462 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.818785 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.818788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.818790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.818924 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.819329 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:13.819334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:13.819544 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 18:01:13.819706 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 18:01:13.819948 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":138,"threads_started":1}
I20260810 18:01:13.822118 31297 heartbeater.cc:347] Connected to a master server at 127.28.83.126:43601
I20260810 18:01:13.822276 31297 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:13.822752 31297 heartbeater.cc:511] Master 127.28.83.126:43601 requested a full tablet report, sending...
I20260810 18:01:13.823812 30795 ts_manager.cc:194] Registered new tserver with Master: cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503)
I20260810 18:01:13.824615 30795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:36171
I20260810 18:01:13.826304 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 31162
I20260810 18:01:13.826421 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:13.826717 29005 external_mini_cluster.cc:1428] Setting key 96e9da1a34ed68c67f1531d6f33291f0
I20260810 18:01:13.828877 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:01:13.832072 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.824504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:33893 (local address 127.28.83.126:43601)
0810 18:01:13.824634 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.824639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.824653 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:01:13.824717 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.824722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.824773 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.824856 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:13.825279 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.825834 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.826969 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.827350 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.827465 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.828658 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:13.828662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:13.828665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:13.828696 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:13.830399 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.830811 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.830818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.830822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.830918 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.831175 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.831181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.831186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.831433 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:13.831539 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.831834 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.831926 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:01:13.832181 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.824250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43601 (local address 127.28.83.67:33893)
0810 18:01:13.824509 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:13.824521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:13.824633 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:13.824871 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:13.824874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:13.825067 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:13.825160 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.825165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.825957 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.825965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.826855 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.826867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.827488 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.827495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.827616 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.828192 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:13.828205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:13.828505 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:13.830521 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.830523 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.830525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.830717 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.831012 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:13.831014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:13.831015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:13.831086 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:13.831548 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:13.831553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:13.831665 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 18:01:13.831783 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 18:01:13.831948 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:01:13 dist-test-slave-srgv krb5kdc[30749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:13.856663 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.848429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.28.83.126:43601)
0810 18:01:13.848594 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.848597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.848610 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:01:13.848652 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.848654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.848694 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.848792 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:13.849553 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.850096 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.850951 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.851174 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.851283 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.853871 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:13.853875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:13.853877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:13.853903 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:13.855168 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.855589 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.855593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.855594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.855680 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:13.855924 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:13.855926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:13.855927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:13.856116 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:13.856219 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.856427 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.856528 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 18:01:13.859759 30795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:13.863157 30795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:13.880211 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.873361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:41275 (local address 127.0.0.1:40476)
0810 18:01:13.874732 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:13.874751 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:13.874914 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:13.875702 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:13.875708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:13.875731 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:13.875864 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.875872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.877295 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.877298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.878681 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.878691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.879668 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.879672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.879786 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.879822 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 18:01:13.879934 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 18:01:13.880056 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"spinlock_wait_cycles":896,"thread_start_us":915,"threads_started":1}
I20260810 18:01:13.880229 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.873689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.28.83.65:41275)
0810 18:01:13.874003 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.874021 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.874834 (+ 813us) server_negotiation.cc:407] Connection header received
0810 18:01:13.875080 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.875090 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.875330 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.875474 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:13.876014 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.877158 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.878828 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.879533 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.879670 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.879823 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.879978 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.880071 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":118,"threads_started":1}
I20260810 18:01:13.883127 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.874880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:34447 (local address 127.0.0.1:44478)
0810 18:01:13.875390 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:13.875405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:13.875495 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:13.876773 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:13.876775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:13.876792 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:13.876873 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.876879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.878260 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.878263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.880017 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.880029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.882629 (+ 2600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.882634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.882741 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.882775 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 18:01:13.882855 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 18:01:13.882944 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":50,"thread_start_us":88,"threads_started":1}
I20260810 18:01:13.883325 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.873279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:41503 (local address 127.0.0.1:39506)
0810 18:01:13.874531 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:13.874584 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:13.874725 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:13.876323 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:13.876326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:13.876346 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:13.876462 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.876469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.878224 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.878227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.879526 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:13.879542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.882924 (+ 3382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.882928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:13.883030 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.883053 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:01:13.883117 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 18:01:13.883188 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":26,"spinlock_wait_cycles":1280,"thread_start_us":121,"threads_started":1}
I20260810 18:01:13.883621 30937 tablet_service.cc:1467] Processing CreateTablet for tablet c7f829cad3d44f18a02a28af6bb6f3e4 (DEFAULT_TABLE table=test-table [id=b2ba3ae9ce1a4dc18859175d6b4f30b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:13.884577 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.875036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.28.83.66:34447)
0810 18:01:13.875330 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.875343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.875609 (+ 266us) server_negotiation.cc:407] Connection header received
0810 18:01:13.875732 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.875737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.876441 (+ 704us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.876597 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:13.876993 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.878128 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.881665 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.882471 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.882595 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.882755 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.884331 (+ 1576us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.884432 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260810 18:01:13.884821 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:13.874228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.28.83.67:41503)
0810 18:01:13.875573 (+ 1345us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:13.875629 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:13.875666 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:01:13.875922 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:13.875929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:13.876147 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:13.876621 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:13.876825 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.877979 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.880171 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:13.882785 (+ 2614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:13.883785 (+ 1000us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:13.883949 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:13.884000 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 18:01:13.884128 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1226,"thread_start_us":108,"threads_started":1}
I20260810 18:01:13.885057 30937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f829cad3d44f18a02a28af6bb6f3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:13.887801 31083 tablet_service.cc:1467] Processing CreateTablet for tablet c7f829cad3d44f18a02a28af6bb6f3e4 (DEFAULT_TABLE table=test-table [id=b2ba3ae9ce1a4dc18859175d6b4f30b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:13.889057 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f829cad3d44f18a02a28af6bb6f3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:13.888967 31229 tablet_service.cc:1467] Processing CreateTablet for tablet c7f829cad3d44f18a02a28af6bb6f3e4 (DEFAULT_TABLE table=test-table [id=b2ba3ae9ce1a4dc18859175d6b4f30b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:13.890084 31229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f829cad3d44f18a02a28af6bb6f3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:13.894078 31321 tablet_bootstrap.cc:492] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Bootstrap starting.
I20260810 18:01:13.896622 31321 tablet_bootstrap.cc:654] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:13.897528 31321 log.cc:824] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:13.898968 31322 tablet_bootstrap.cc:492] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: Bootstrap starting.
I20260810 18:01:13.899618 31321 tablet_bootstrap.cc:492] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: No bootstrap required, opened a new log
I20260810 18:01:13.899827 31321 ts_tablet_manager.cc:1397] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:01:13.901278 31322 tablet_bootstrap.cc:654] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:13.902181 31322 log.cc:824] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:13.902891 31324 tablet_bootstrap.cc:492] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: Bootstrap starting.
I20260810 18:01:13.903966 31322 tablet_bootstrap.cc:492] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: No bootstrap required, opened a new log
I20260810 18:01:13.904186 31322 ts_tablet_manager.cc:1397] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:01:13.904232 31321 raft_consensus.cc:348] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } }
I20260810 18:01:13.904515 31321 raft_consensus.cc:374] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:13.904595 31321 raft_consensus.cc:729] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd0d725e3d24fe7a7c3d278d5bf5005, State: Initialized, Role: FOLLOWER
I20260810 18:01:13.905058 31321 consensus_queue.cc:260] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } }
I20260810 18:01:13.905156 31324 tablet_bootstrap.cc:654] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:13.905995 31324 log.cc:824] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:13.906278 31005 heartbeater.cc:503] Master 127.28.83.126:43601 was elected leader, sending a full tablet report...
I20260810 18:01:13.908129 31324 tablet_bootstrap.cc:492] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: No bootstrap required, opened a new log
I20260810 18:01:13.908329 31324 ts_tablet_manager.cc:1397] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260810 18:01:13.908443 31321 ts_tablet_manager.cc:1428] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 18:01:13.908298 31322 raft_consensus.cc:348] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } }
I20260810 18:01:13.908563 31322 raft_consensus.cc:374] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:13.908613 31322 raft_consensus.cc:729] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2dc2e3ccb046308e78478bba960790, State: Initialized, Role: FOLLOWER
I20260810 18:01:13.909055 31322 consensus_queue.cc:260] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } }
I20260810 18:01:13.909808 31297 heartbeater.cc:503] Master 127.28.83.126:43601 was elected leader, sending a full tablet report...
I20260810 18:01:13.910037 31322 ts_tablet_manager.cc:1428] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:01:13.912441 31324 raft_consensus.cc:348] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } }
I20260810 18:01:13.912726 31324 raft_consensus.cc:374] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:13.912802 31324 raft_consensus.cc:729] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26a8527c619486ca2f15ef8189e210a, State: Initialized, Role: FOLLOWER
I20260810 18:01:13.913286 31324 consensus_queue.cc:260] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } }
I20260810 18:01:13.914306 31324 ts_tablet_manager.cc:1428] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:01:13.914702 31153 heartbeater.cc:503] Master 127.28.83.126:43601 was elected leader, sending a full tablet report...
I20260810 18:01:14.067989 31326 raft_consensus.cc:482] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:14.068255 31326 raft_consensus.cc:504] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } }
W20260810 18:01:14.068724 31298 tablet.cc:2369] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:14.069638 31326 leader_election.cc:288] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503), d26a8527c619486ca2f15ef8189e210a (127.28.83.66:34447)
I20260810 18:01:14.074440 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.070041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:43289 (local address 127.28.83.67:41503)
0810 18:01:14.070198 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.070204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.070425 (+ 221us) server_negotiation.cc:407] Connection header received
0810 18:01:14.070617 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.070620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.070672 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.070762 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:14.071037 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.071560 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.073226 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.073807 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.073924 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.073963 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.074230 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.074306 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 18:01:14.074440 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.070083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:59863 (local address 127.28.83.66:34447)
0810 18:01:14.070229 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.070235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.070545 (+ 310us) server_negotiation.cc:407] Connection header received
0810 18:01:14.070703 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.070706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.070755 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.070838 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:14.071104 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.071634 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.073160 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.073713 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.073842 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.073924 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.074231 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.074306 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:01:14.074550 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.069967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:34447 (local address 127.28.83.65:59863)
0810 18:01:14.070443 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:14.070455 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:14.070596 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:14.070860 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:14.070863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:14.070878 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:14.070996 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.071001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.071767 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.071770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.073011 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.073021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.073971 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.073974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.074108 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.074125 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 18:01:14.074193 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:01:14.074346 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":90,"thread_start_us":99,"threads_started":1}
I20260810 18:01:14.074550 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.069915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:41503 (local address 127.28.83.65:43289)
0810 18:01:14.070326 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:14.070338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:14.070501 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:14.070777 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:14.070780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:14.070796 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:14.070927 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.070933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.071682 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.071685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.073081 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.073117 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.073938 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.073944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.074082 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.074109 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:01:14.074194 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:01:14.074347 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":210,"thread_start_us":145,"threads_started":1}
I20260810 18:01:14.075225 31102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f829cad3d44f18a02a28af6bb6f3e4" candidate_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26a8527c619486ca2f15ef8189e210a" is_pre_election: true
I20260810 18:01:14.075274 31250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f829cad3d44f18a02a28af6bb6f3e4" candidate_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790" is_pre_election: true
I20260810 18:01:14.075556 31250 raft_consensus.cc:2454] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd0d725e3d24fe7a7c3d278d5bf5005 in term 0.
I20260810 18:01:14.075552 31102 raft_consensus.cc:2454] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd0d725e3d24fe7a7c3d278d5bf5005 in term 0.
I20260810 18:01:14.076087 30887 leader_election.cc:302] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, cf2dc2e3ccb046308e78478bba960790, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:14.076406 31326 raft_consensus.cc:2781] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:14.076514 31326 raft_consensus.cc:482] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:14.076576 31326 raft_consensus.cc:3037] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:14.077716 31326 raft_consensus.cc:504] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } }
I20260810 18:01:14.078094 31326 leader_election.cc:288] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [CANDIDATE]: Term 1 election: Requested vote from peers cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503), d26a8527c619486ca2f15ef8189e210a (127.28.83.66:34447)
I20260810 18:01:14.078508 31250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f829cad3d44f18a02a28af6bb6f3e4" candidate_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790"
I20260810 18:01:14.078532 31102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f829cad3d44f18a02a28af6bb6f3e4" candidate_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26a8527c619486ca2f15ef8189e210a"
I20260810 18:01:14.078646 31250 raft_consensus.cc:3037] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:14.078666 31102 raft_consensus.cc:3037] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:14.079766 31102 raft_consensus.cc:2454] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd0d725e3d24fe7a7c3d278d5bf5005 in term 1.
I20260810 18:01:14.080013 31250 raft_consensus.cc:2454] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd0d725e3d24fe7a7c3d278d5bf5005 in term 1.
I20260810 18:01:14.080147 30887 leader_election.cc:302] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:14.080349 31326 raft_consensus.cc:2781] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:14.080588 31326 raft_consensus.cc:686] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd0d725e3d24fe7a7c3d278d5bf5005, State: Running, Role: LEADER
I20260810 18:01:14.080957 31326 consensus_queue.cc:237] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } }
I20260810 18:01:14.084429 30795 catalog_manager.cc:5495] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65). New cstate: current_term: 1 leader_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } health_report { overall_health: UNKNOWN } } }
W20260810 18:01:14.092724 31154 tablet.cc:2369] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:01:14.122293 31006 tablet.cc:2369] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 18:01:14 dist-test-slave-srgv krb5kdc[30749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:14.148391 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.139316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43601 (local address 127.0.0.1:35524)
0810 18:01:14.139628 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:14.139640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:14.139810 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:14.140060 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:14.140063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:14.140508 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:14.140628 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.140638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.141474 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.141480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.142183 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.142193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.142644 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.142647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.142739 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.143291 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:14.143303 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:14.144797 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:14.146722 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:14.146731 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:14.146738 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:14.147004 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:14.147301 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:14.147303 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:14.147305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:14.147388 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:14.147781 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:14.147791 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:14.147900 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 18:01:14.148015 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 18:01:14.148184 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260810 18:01:14.148391 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.139363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.28.83.126:43601)
0810 18:01:14.139510 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.139516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.139746 (+ 230us) server_negotiation.cc:407] Connection header received
0810 18:01:14.139909 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.139912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.139956 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.140043 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:14.140756 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.141346 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.142305 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.142534 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.142640 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.144935 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:14.144938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:14.144941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:14.144969 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:14.146603 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:14.147105 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:14.147111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:14.147113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:14.147189 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:14.147474 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:14.147481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:14.147482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:14.147664 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:14.147768 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.148061 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.148259 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":45}
I20260810 18:01:14.152487 30795 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:35524:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:01:14.153301 30795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:14.158396 31229 tablet_service.cc:1467] Processing CreateTablet for tablet 5f01094f4ed64c35aec7c0b2d081768a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e18410509b5043e4adc29f0688c92340]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:01:14.158221 30937 tablet_service.cc:1467] Processing CreateTablet for tablet 5f01094f4ed64c35aec7c0b2d081768a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e18410509b5043e4adc29f0688c92340]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:01:14.158226 31083 tablet_service.cc:1467] Processing CreateTablet for tablet 5f01094f4ed64c35aec7c0b2d081768a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e18410509b5043e4adc29f0688c92340]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:01:14.159009 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f01094f4ed64c35aec7c0b2d081768a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:14.159009 31229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f01094f4ed64c35aec7c0b2d081768a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:14.159025 30937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f01094f4ed64c35aec7c0b2d081768a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:14.161899 31321 tablet_bootstrap.cc:492] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: Bootstrap starting.
I20260810 18:01:14.161943 31324 tablet_bootstrap.cc:492] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: Bootstrap starting.
I20260810 18:01:14.162647 31322 tablet_bootstrap.cc:492] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: Bootstrap starting.
I20260810 18:01:14.162967 31324 tablet_bootstrap.cc:654] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:14.163285 31321 tablet_bootstrap.cc:654] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:14.163763 31322 tablet_bootstrap.cc:654] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:14.164155 31324 tablet_bootstrap.cc:492] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: No bootstrap required, opened a new log
I20260810 18:01:14.164245 31324 ts_tablet_manager.cc:1397] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:14.164592 31321 tablet_bootstrap.cc:492] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: No bootstrap required, opened a new log
I20260810 18:01:14.164678 31321 ts_tablet_manager.cc:1397] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:14.164736 31324 raft_consensus.cc:348] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.164865 31324 raft_consensus.cc:374] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:14.164906 31324 raft_consensus.cc:729] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26a8527c619486ca2f15ef8189e210a, State: Initialized, Role: FOLLOWER
I20260810 18:01:14.165036 31324 consensus_queue.cc:260] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.165139 31321 raft_consensus.cc:348] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.165346 31324 ts_tablet_manager.cc:1428] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 18:01:14.165364 31329 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.165541 31329 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:14.165617 31322 tablet_bootstrap.cc:492] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: No bootstrap required, opened a new log
I20260810 18:01:14.165241 31321 raft_consensus.cc:374] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:14.165702 31322 ts_tablet_manager.cc:1397] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:14.165726 31321 raft_consensus.cc:729] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd0d725e3d24fe7a7c3d278d5bf5005, State: Initialized, Role: FOLLOWER
I20260810 18:01:14.165881 31321 consensus_queue.cc:260] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.166204 31321 ts_tablet_manager.cc:1428] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:14.166149 31326 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.166196 31322 raft_consensus.cc:348] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.166277 31326 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:14.166308 31322 raft_consensus.cc:374] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:14.166355 31322 raft_consensus.cc:729] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2dc2e3ccb046308e78478bba960790, State: Initialized, Role: FOLLOWER
I20260810 18:01:14.166487 31322 consensus_queue.cc:260] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.166790 31322 ts_tablet_manager.cc:1428] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:01:14.166792 31328 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.167282 31328 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:14.271661 31326 raft_consensus.cc:482] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:14.271817 31326 raft_consensus.cc:504] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.272360 31326 leader_election.cc:288] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d26a8527c619486ca2f15ef8189e210a (127.28.83.66:34447), cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503)
I20260810 18:01:14.272871 31102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f01094f4ed64c35aec7c0b2d081768a" candidate_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26a8527c619486ca2f15ef8189e210a" is_pre_election: true
I20260810 18:01:14.272943 31250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f01094f4ed64c35aec7c0b2d081768a" candidate_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790" is_pre_election: true
I20260810 18:01:14.273036 31102 raft_consensus.cc:2454] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd0d725e3d24fe7a7c3d278d5bf5005 in term 0.
I20260810 18:01:14.273073 31250 raft_consensus.cc:2454] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd0d725e3d24fe7a7c3d278d5bf5005 in term 0.
I20260810 18:01:14.273358 30887 leader_election.cc:302] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:14.273604 31326 raft_consensus.cc:2781] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:14.273672 31326 raft_consensus.cc:482] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:14.273718 31326 raft_consensus.cc:3037] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:14.274718 31326 raft_consensus.cc:504] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.275054 31326 leader_election.cc:288] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [CANDIDATE]: Term 1 election: Requested vote from peers d26a8527c619486ca2f15ef8189e210a (127.28.83.66:34447), cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503)
I20260810 18:01:14.275457 31102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f01094f4ed64c35aec7c0b2d081768a" candidate_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d26a8527c619486ca2f15ef8189e210a"
I20260810 18:01:14.275480 31250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f01094f4ed64c35aec7c0b2d081768a" candidate_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790"
I20260810 18:01:14.275594 31102 raft_consensus.cc:3037] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:14.275594 31250 raft_consensus.cc:3037] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:14.276749 31250 raft_consensus.cc:2454] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd0d725e3d24fe7a7c3d278d5bf5005 in term 1.
I20260810 18:01:14.276755 31102 raft_consensus.cc:2454] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd0d725e3d24fe7a7c3d278d5bf5005 in term 1.
I20260810 18:01:14.277032 30887 leader_election.cc:302] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:14.277232 31326 raft_consensus.cc:2781] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:14.277344 31326 raft_consensus.cc:686] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd0d725e3d24fe7a7c3d278d5bf5005, State: Running, Role: LEADER
I20260810 18:01:14.277515 31326 consensus_queue.cc:237] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.278246 31332 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: TxnStatusTablet state changed. Reason: New leader 5dd0d725e3d24fe7a7c3d278d5bf5005. Latest consensus state: current_term: 1 leader_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.278369 31332 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:01:14.278707 31345 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:01:14.279173 31345 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:01:14.279438 30794 catalog_manager.cc:5495] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65). New cstate: current_term: 1 leader_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } health_report { overall_health: UNKNOWN } } }
I20260810 18:01:14.319418 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.315202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.28.83.65:41275)
0810 18:01:14.315333 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.315338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.315354 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:01:14.315502 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.315506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.315585 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.315694 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:14.315922 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.316753 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.317819 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.318178 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.318292 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.318533 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.319058 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.319175 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.319219 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.319279 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:01:14.319445 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.315076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:41275 (local address 127.0.0.1:40492)
0810 18:01:14.315241 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:14.315257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:14.315353 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:14.315710 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:14.315712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:14.315718 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:14.315819 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.315823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.316873 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.316875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.317692 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.317704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.318287 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.318289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.318353 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.318396 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.319151 (+ 755us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.319161 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 18:01:14.319208 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 18:01:14.319278 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 18:01:14.324995 31102 raft_consensus.cc:1261] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Refusing update from remote peer 5dd0d725e3d24fe7a7c3d278d5bf5005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:14.325042 31250 raft_consensus.cc:1261] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Refusing update from remote peer 5dd0d725e3d24fe7a7c3d278d5bf5005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:14.325779 31326 consensus_queue.cc:1035] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:14.325989 31332 consensus_queue.cc:1035] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:14.330150 31328 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: TxnStatusTablet state changed. Reason: New leader 5dd0d725e3d24fe7a7c3d278d5bf5005. Latest consensus state: current_term: 1 leader_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.330149 31329 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: TxnStatusTablet state changed. Reason: New leader 5dd0d725e3d24fe7a7c3d278d5bf5005. Latest consensus state: current_term: 1 leader_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.330283 31329 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:14.330281 31328 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:14.333267 31326 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.333389 31326 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:01:14.333514 31326 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.333595 31326 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P 5dd0d725e3d24fe7a7c3d278d5bf5005: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:01:14.333900 31348 mvcc.cc:205] Tried to move back new op lower bound from 7317032445228347392 to 7317032445042135040. Current Snapshot: MvccSnapshot[applied={T|T < 7317032445228347392}]
I20260810 18:01:14.334342 31328 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.334447 31328 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P cf2dc2e3ccb046308e78478bba960790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:14.334524 31329 tablet_replica.cc:439] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } } }
I20260810 18:01:14.334632 31329 tablet_replica.cc:442] T 5f01094f4ed64c35aec7c0b2d081768a P d26a8527c619486ca2f15ef8189e210a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:14.336092 31350 mvcc.cc:205] Tried to move back new op lower bound from 7317032445228347392 to 7317032445042135040. Current Snapshot: MvccSnapshot[applied={T|T < 7317032445228347392}]
I20260810 18:01:14.338220 31349 mvcc.cc:205] Tried to move back new op lower bound from 7317032445228347392 to 7317032445042135040. Current Snapshot: MvccSnapshot[applied={T|T < 7317032445228347392}]
I20260810 18:01:14.346648 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.342999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.28.83.65:41275)
0810 18:01:14.343313 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.343317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.343329 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:01:14.343370 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.343373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.343421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.343494 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:14.343812 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.344601 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.345470 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.345821 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.345924 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.346173 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.346256 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.346326 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.346447 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.346499 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260810 18:01:14.357828 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.354481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:47843 (local address 127.28.83.65:41275)
0810 18:01:14.354599 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.354603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.354776 (+ 173us) server_negotiation.cc:407] Connection header received
0810 18:01:14.354924 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.354929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.354977 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.355059 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:14.355367 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.355857 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.356719 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.356925 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.357032 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.357328 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.357399 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.357526 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.357612 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.357660 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":39}
I20260810 18:01:14.357838 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.354403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:41275 (local address 127.28.83.65:47843)
0810 18:01:14.354684 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:14.354697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:14.354826 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:14.355085 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:14.355088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:14.355094 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:14.355268 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.355274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.355965 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.355970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.356588 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.356595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.357033 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.357036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.357124 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.357170 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.357539 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.357546 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:01:14.357605 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 18:01:14.357680 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":23,"thread_start_us":91,"threads_started":1}
I20260810 18:01:14.364662 31250 raft_consensus.cc:1261] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Refusing update from remote peer 5dd0d725e3d24fe7a7c3d278d5bf5005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:14.364777 31102 raft_consensus.cc:1261] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Refusing update from remote peer 5dd0d725e3d24fe7a7c3d278d5bf5005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:14.365180 31361 consensus_queue.cc:1035] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:14.365391 31332 consensus_queue.cc:1035] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:14.409143 30794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:14.409309 30794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:14.411755 30794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989: Sending DeleteTablet for 3 replicas of tablet c7f829cad3d44f18a02a28af6bb6f3e4
I20260810 18:01:14.412488 31229 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f829cad3d44f18a02a28af6bb6f3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:39506
I20260810 18:01:14.412586 31083 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f829cad3d44f18a02a28af6bb6f3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:44478
I20260810 18:01:14.412995 31366 tablet_replica.cc:330] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: stopping tablet replica
I20260810 18:01:14.413077 30936 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f829cad3d44f18a02a28af6bb6f3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:40476
I20260810 18:01:14.413297 31365 tablet_replica.cc:330] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: stopping tablet replica
I20260810 18:01:14.413563 31367 tablet_replica.cc:330] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: stopping tablet replica
I20260810 18:01:14.413612 30794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:01:14.413846 31367 raft_consensus.cc:2229] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 LEADER]: Raft consensus shutting down.
W20260810 18:01:14.414029 30794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:14.414232 31367 raft_consensus.cc:2258] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:14.414472 31365 raft_consensus.cc:2229] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:14.414680 31366 raft_consensus.cc:2229] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:14.415026 31366 raft_consensus.cc:2258] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:14.415313 31365 raft_consensus.cc:2258] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:14.415551 31367 ts_tablet_manager.cc:1905] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:14.417392 31365 ts_tablet_manager.cc:1905] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:14.419600 30936 tablet_service.cc:1467] Processing CreateTablet for tablet f4c6096afe7348f0b11e90ceda345c57 (DEFAULT_TABLE table=test-table [id=0e51299a3cc34615894b18cc206a2c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:14.419970 31229 tablet_service.cc:1467] Processing CreateTablet for tablet f4c6096afe7348f0b11e90ceda345c57 (DEFAULT_TABLE table=test-table [id=0e51299a3cc34615894b18cc206a2c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:14.420042 30936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4c6096afe7348f0b11e90ceda345c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:14.420228 31229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4c6096afe7348f0b11e90ceda345c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:14.421015 31365 ts_tablet_manager.cc:1918] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:14.421178 31365 log.cc:1196] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/c7f829cad3d44f18a02a28af6bb6f3e4
I20260810 18:01:14.421722 31365 ts_tablet_manager.cc:1939] T c7f829cad3d44f18a02a28af6bb6f3e4 P cf2dc2e3ccb046308e78478bba960790: Deleting consensus metadata
I20260810 18:01:14.423184 30781 catalog_manager.cc:4841] TS cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503): tablet c7f829cad3d44f18a02a28af6bb6f3e4 (table test-table [id=b2ba3ae9ce1a4dc18859175d6b4f30b4]) successfully deleted
I20260810 18:01:14.423769 31321 tablet_bootstrap.cc:492] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Bootstrap starting.
I20260810 18:01:14.423841 31322 tablet_bootstrap.cc:492] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: Bootstrap starting.
I20260810 18:01:14.423910 31083 tablet_service.cc:1467] Processing CreateTablet for tablet f4c6096afe7348f0b11e90ceda345c57 (DEFAULT_TABLE table=test-table [id=0e51299a3cc34615894b18cc206a2c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:14.424217 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4c6096afe7348f0b11e90ceda345c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:14.425125 31322 tablet_bootstrap.cc:654] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:14.425846 31321 tablet_bootstrap.cc:654] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:14.427208 31324 tablet_bootstrap.cc:492] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: Bootstrap starting.
I20260810 18:01:14.429364 31324 tablet_bootstrap.cc:654] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:14.432663 31321 tablet_bootstrap.cc:492] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: No bootstrap required, opened a new log
I20260810 18:01:14.432812 31321 ts_tablet_manager.cc:1397] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260810 18:01:14.433760 31321 raft_consensus.cc:348] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.433888 31321 raft_consensus.cc:374] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:14.433925 31321 raft_consensus.cc:729] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd0d725e3d24fe7a7c3d278d5bf5005, State: Initialized, Role: FOLLOWER
I20260810 18:01:14.434122 31321 consensus_queue.cc:260] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.435205 31366 ts_tablet_manager.cc:1905] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:14.436743 31321 ts_tablet_manager.cc:1428] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 18:01:14.439402 31367 ts_tablet_manager.cc:1918] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:14.439472 31366 ts_tablet_manager.cc:1918] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:14.439519 31367 log.cc:1196] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/c7f829cad3d44f18a02a28af6bb6f3e4
I20260810 18:01:14.439584 31366 log.cc:1196] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/c7f829cad3d44f18a02a28af6bb6f3e4
I20260810 18:01:14.439940 31367 ts_tablet_manager.cc:1939] T c7f829cad3d44f18a02a28af6bb6f3e4 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Deleting consensus metadata
I20260810 18:01:14.439945 31366 ts_tablet_manager.cc:1939] T c7f829cad3d44f18a02a28af6bb6f3e4 P d26a8527c619486ca2f15ef8189e210a: Deleting consensus metadata
I20260810 18:01:14.441164 30779 catalog_manager.cc:4841] TS 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275): tablet c7f829cad3d44f18a02a28af6bb6f3e4 (table test-table [id=b2ba3ae9ce1a4dc18859175d6b4f30b4]) successfully deleted
I20260810 18:01:14.441484 30779 catalog_manager.cc:4841] TS d26a8527c619486ca2f15ef8189e210a (127.28.83.66:34447): tablet c7f829cad3d44f18a02a28af6bb6f3e4 (table test-table [id=b2ba3ae9ce1a4dc18859175d6b4f30b4]) successfully deleted
I20260810 18:01:14.441766 31324 tablet_bootstrap.cc:492] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: No bootstrap required, opened a new log
I20260810 18:01:14.441879 31324 ts_tablet_manager.cc:1397] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20260810 18:01:14.442360 31324 raft_consensus.cc:348] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.442495 31324 raft_consensus.cc:374] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:14.442529 31324 raft_consensus.cc:729] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26a8527c619486ca2f15ef8189e210a, State: Initialized, Role: FOLLOWER
I20260810 18:01:14.442699 31324 consensus_queue.cc:260] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.442751 31322 tablet_bootstrap.cc:492] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: No bootstrap required, opened a new log
I20260810 18:01:14.442847 31322 ts_tablet_manager.cc:1397] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20260810 18:01:14.443014 31324 ts_tablet_manager.cc:1428] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:14.443532 31322 raft_consensus.cc:348] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.443681 31322 raft_consensus.cc:374] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:14.443725 31322 raft_consensus.cc:729] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2dc2e3ccb046308e78478bba960790, State: Initialized, Role: FOLLOWER
I20260810 18:01:14.443885 31322 consensus_queue.cc:260] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.444809 31322 ts_tablet_manager.cc:1428] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 18:01:14.595450 31329 raft_consensus.cc:482] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:14.595600 31329 raft_consensus.cc:504] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.596884 31329 leader_election.cc:288] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275), cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503)
I20260810 18:01:14.602221 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.597122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:41275 (local address 127.28.83.66:44667)
0810 18:01:14.597609 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:14.597626 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:14.597773 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:14.598119 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:14.598121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:14.598136 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:14.598316 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.598327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.599422 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.599426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.600760 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.600771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.601794 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.601797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.601893 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.601916 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:01:14.601981 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:01:14.602045 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":61,"thread_start_us":71,"threads_started":1}
I20260810 18:01:14.602631 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.597493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:44667 (local address 127.28.83.65:41275)
0810 18:01:14.597609 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.597614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.597691 (+ 77us) server_negotiation.cc:407] Connection header received
0810 18:01:14.597884 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.597888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.597929 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.597990 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:14.598474 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.599275 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.600904 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.601568 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.601679 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.601732 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.602438 (+ 706us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.602520 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:01:14.603300 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.597269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:41503 (local address 127.28.83.66:51033)
0810 18:01:14.598567 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:14.598579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:14.598647 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:14.599003 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:14.599006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:14.599021 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:14.599162 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.599167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.600176 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.600180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.601398 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.601407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.602830 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.602836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.602960 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.602982 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:01:14.603056 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 18:01:14.603118 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260810 18:01:14.603241 30958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c6096afe7348f0b11e90ceda345c57" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" is_pre_election: true
I20260810 18:01:14.603462 30958 raft_consensus.cc:2454] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 0.
I20260810 18:01:14.603826 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.597617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:51033 (local address 127.28.83.67:41503)
0810 18:01:14.597916 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.597920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.598732 (+ 812us) server_negotiation.cc:407] Connection header received
0810 18:01:14.598789 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.598791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.598836 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.598906 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:14.599279 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.600060 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.601558 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.602250 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.602578 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.602628 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.603591 (+ 963us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.603676 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260810 18:01:14.603881 31032 leader_election.cc:302] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:14.604525 31329 raft_consensus.cc:2781] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:14.604575 31329 raft_consensus.cc:482] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:14.604538 31249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c6096afe7348f0b11e90ceda345c57" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790" is_pre_election: true
I20260810 18:01:14.604610 31329 raft_consensus.cc:3037] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:14.604671 31249 raft_consensus.cc:2454] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 0.
I20260810 18:01:14.606395 31329 raft_consensus.cc:504] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.606855 31329 leader_election.cc:288] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 election: Requested vote from peers 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275), cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503)
I20260810 18:01:14.607167 30958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c6096afe7348f0b11e90ceda345c57" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005"
I20260810 18:01:14.607285 30958 raft_consensus.cc:3037] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:14.607265 31249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c6096afe7348f0b11e90ceda345c57" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790"
I20260810 18:01:14.607365 31249 raft_consensus.cc:3037] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:14.608866 30958 raft_consensus.cc:2454] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 1.
I20260810 18:01:14.608893 31249 raft_consensus.cc:2454] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 1.
I20260810 18:01:14.609166 31032 leader_election.cc:302] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:14.609357 31329 raft_consensus.cc:2781] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:14.609588 31329 raft_consensus.cc:686] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 1 LEADER]: Becoming Leader. State: Replica: d26a8527c619486ca2f15ef8189e210a, State: Running, Role: LEADER
I20260810 18:01:14.609828 31329 consensus_queue.cc:237] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:14.612438 30794 catalog_manager.cc:5495] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a reported cstate change: term changed from 0 to 1, leader changed from <none> to d26a8527c619486ca2f15ef8189e210a (127.28.83.66). New cstate: current_term: 1 leader_uuid: "d26a8527c619486ca2f15ef8189e210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } health_report { overall_health: UNKNOWN } } }
I20260810 18:01:14.643930 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.639933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.28.83.66:34447)
0810 18:01:14.640314 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.640362 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.640375 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:01:14.640443 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.640445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.640496 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.640659 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:14.640881 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.641647 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.642462 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.642751 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.642832 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.643064 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.643560 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.643646 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.643700 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.643747 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":169,"threads_started":1}
I20260810 18:01:14.655177 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.652113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:56459 (local address 127.28.83.65:41275)
0810 18:01:14.652236 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.652239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.652359 (+ 120us) server_negotiation.cc:407] Connection header received
0810 18:01:14.652467 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.652469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.652513 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.652575 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:14.652900 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.653393 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.654263 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.654472 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.654549 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.654808 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.654872 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.654937 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.655027 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.655068 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:01:14.655268 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.651888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:41275 (local address 127.28.83.66:56459)
0810 18:01:14.652283 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:14.652296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:14.652386 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:14.652597 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:14.652598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:14.652604 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:14.652795 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.652800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.653529 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.653531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.654155 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.654163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.654566 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.654568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.654655 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.654694 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.654953 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.654957 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:01:14.655003 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 18:01:14.655063 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":177,"threads_started":1}
I20260810 18:01:14.661491 31249 raft_consensus.cc:1261] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Refusing update from remote peer d26a8527c619486ca2f15ef8189e210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:14.661733 30958 raft_consensus.cc:1261] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Refusing update from remote peer d26a8527c619486ca2f15ef8189e210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:14.662025 31374 consensus_queue.cc:1035] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:14.662245 31329 consensus_queue.cc:1035] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:14.679860 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.676523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:34447 (local address 127.28.83.65:40299)
0810 18:01:14.676735 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:14.676749 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:14.676837 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:14.677064 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:14.677067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:14.677073 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:14.677173 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.677179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.678089 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.678093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.678675 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:14.678683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.679161 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.679164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:14.679250 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.679286 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.679570 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.679578 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:01:14.679640 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:01:14.679749 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 18:01:14.679850 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:14.676588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:40299 (local address 127.28.83.66:34447)
0810 18:01:14.676702 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:14.676707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:14.676808 (+ 101us) server_negotiation.cc:407] Connection header received
0810 18:01:14.676917 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:14.676920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:14.676963 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:14.677044 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:14.677296 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.677961 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.678816 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:14.679052 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:14.679161 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:14.679401 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:14.679490 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:14.679566 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:14.679650 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 18:01:14.679701 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 18:01:14.680896 31350 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 18:01:16.242836 30794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:16.243034 30794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:16.245647 30794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989: Sending DeleteTablet for 3 replicas of tablet f4c6096afe7348f0b11e90ceda345c57
I20260810 18:01:16.246304 30936 tablet_service.cc:1514] Processing DeleteTablet for tablet f4c6096afe7348f0b11e90ceda345c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:16 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:40476
I20260810 18:01:16.246407 31229 tablet_service.cc:1514] Processing DeleteTablet for tablet f4c6096afe7348f0b11e90ceda345c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:16 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:39506
I20260810 18:01:16.246822 31083 tablet_service.cc:1514] Processing DeleteTablet for tablet f4c6096afe7348f0b11e90ceda345c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:16 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:44478
I20260810 18:01:16.247692 31408 tablet_replica.cc:330] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: stopping tablet replica
I20260810 18:01:16.247869 31408 raft_consensus.cc:2229] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:16.248191 31408 raft_consensus.cc:2258] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:16.249174 31408 ts_tablet_manager.cc:1905] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:16.249969 31406 tablet_replica.cc:330] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: stopping tablet replica
I20260810 18:01:16.250128 31406 raft_consensus.cc:2229] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:16.250274 31406 raft_consensus.cc:2258] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:16.250560 31407 tablet_replica.cc:330] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: stopping tablet replica
I20260810 18:01:16.250715 31407 raft_consensus.cc:2229] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:16.250857 31407 raft_consensus.cc:2258] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:16.251185 31406 ts_tablet_manager.cc:1905] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:16.251576 31407 ts_tablet_manager.cc:1905] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:16.252815 31408 ts_tablet_manager.cc:1918] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:01:16.252916 31408 log.cc:1196] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/f4c6096afe7348f0b11e90ceda345c57
I20260810 18:01:16.253257 31408 ts_tablet_manager.cc:1939] T f4c6096afe7348f0b11e90ceda345c57 P d26a8527c619486ca2f15ef8189e210a: Deleting consensus metadata
I20260810 18:01:16.254113 30779 catalog_manager.cc:4841] TS d26a8527c619486ca2f15ef8189e210a (127.28.83.66:34447): tablet f4c6096afe7348f0b11e90ceda345c57 (table test-table [id=0e51299a3cc34615894b18cc206a2c93]) successfully deleted
I20260810 18:01:16.254527 31407 ts_tablet_manager.cc:1918] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:01:16.254635 31407 log.cc:1196] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/f4c6096afe7348f0b11e90ceda345c57
I20260810 18:01:16.254727 31406 ts_tablet_manager.cc:1918] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:01:16.254842 31406 log.cc:1196] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/f4c6096afe7348f0b11e90ceda345c57
I20260810 18:01:16.254951 31407 ts_tablet_manager.cc:1939] T f4c6096afe7348f0b11e90ceda345c57 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Deleting consensus metadata
I20260810 18:01:16.255160 31406 ts_tablet_manager.cc:1939] T f4c6096afe7348f0b11e90ceda345c57 P cf2dc2e3ccb046308e78478bba960790: Deleting consensus metadata
I20260810 18:01:16.255721 30779 catalog_manager.cc:4841] TS 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275): tablet f4c6096afe7348f0b11e90ceda345c57 (table test-table [id=0e51299a3cc34615894b18cc206a2c93]) successfully deleted
I20260810 18:01:16.255959 30781 catalog_manager.cc:4841] TS cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503): tablet f4c6096afe7348f0b11e90ceda345c57 (table test-table [id=0e51299a3cc34615894b18cc206a2c93]) successfully deleted
Aug 10 18:01:16 dist-test-slave-srgv krb5kdc[30749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:01:16 dist-test-slave-srgv krb5kdc[30749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:16.284613 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:16.275194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.28.83.126:43601)
0810 18:01:16.275608 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:16.275655 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:16.275673 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:01:16.275761 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:16.275764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:16.275810 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:16.275901 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:16.276697 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:16.277315 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:16.278194 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:16.278418 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:16.278498 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:16.281114 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:16.281118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:16.281121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:16.281154 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:16.282853 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:16.283303 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:16.283307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:16.283308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:16.283394 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:16.283685 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:16.283689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:16.283691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:16.283879 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:16.284023 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:16.284294 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 18:01:16.284426 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":197,"threads_started":1}
I20260810 18:01:16.287452 30794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:16.287925 30794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:16.292831 30936 tablet_service.cc:1467] Processing CreateTablet for tablet 8df029cff1fb442c88881304946f5e64 (DEFAULT_TABLE table=test-table [id=8bf573f108374e7181d7bbf69e76064a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:16.292865 31083 tablet_service.cc:1467] Processing CreateTablet for tablet 8df029cff1fb442c88881304946f5e64 (DEFAULT_TABLE table=test-table [id=8bf573f108374e7181d7bbf69e76064a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:16.293044 31229 tablet_service.cc:1467] Processing CreateTablet for tablet 8df029cff1fb442c88881304946f5e64 (DEFAULT_TABLE table=test-table [id=8bf573f108374e7181d7bbf69e76064a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:16.293197 30936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df029cff1fb442c88881304946f5e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:16.293198 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df029cff1fb442c88881304946f5e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:16.293319 31229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df029cff1fb442c88881304946f5e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:16.295406 31417 tablet_bootstrap.cc:492] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: Bootstrap starting.
I20260810 18:01:16.296254 31419 tablet_bootstrap.cc:492] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: Bootstrap starting.
I20260810 18:01:16.296327 31417 tablet_bootstrap.cc:654] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:16.296876 31418 tablet_bootstrap.cc:492] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Bootstrap starting.
I20260810 18:01:16.297622 31419 tablet_bootstrap.cc:654] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:16.298017 31418 tablet_bootstrap.cc:654] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:16.298264 31417 tablet_bootstrap.cc:492] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: No bootstrap required, opened a new log
I20260810 18:01:16.298379 31417 ts_tablet_manager.cc:1397] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:16.298897 31417 raft_consensus.cc:348] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.299026 31417 raft_consensus.cc:374] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:16.299067 31417 raft_consensus.cc:729] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26a8527c619486ca2f15ef8189e210a, State: Initialized, Role: FOLLOWER
I20260810 18:01:16.299238 31419 tablet_bootstrap.cc:492] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: No bootstrap required, opened a new log
I20260810 18:01:16.299218 31417 consensus_queue.cc:260] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.299404 31419 ts_tablet_manager.cc:1397] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:16.299686 31417 ts_tablet_manager.cc:1428] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 18:01:16.299777 31418 tablet_bootstrap.cc:492] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: No bootstrap required, opened a new log
I20260810 18:01:16.299880 31418 ts_tablet_manager.cc:1397] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:16.300055 31419 raft_consensus.cc:348] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.300149 31419 raft_consensus.cc:374] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:16.300172 31419 raft_consensus.cc:729] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2dc2e3ccb046308e78478bba960790, State: Initialized, Role: FOLLOWER
I20260810 18:01:16.300330 31419 consensus_queue.cc:260] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.300484 31418 raft_consensus.cc:348] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.300609 31418 raft_consensus.cc:374] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:16.300666 31418 raft_consensus.cc:729] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd0d725e3d24fe7a7c3d278d5bf5005, State: Initialized, Role: FOLLOWER
I20260810 18:01:16.300822 31418 consensus_queue.cc:260] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.300961 31419 ts_tablet_manager.cc:1428] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:16.301121 31418 ts_tablet_manager.cc:1428] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:16.378794 31400 raft_consensus.cc:482] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:16.378975 31400 raft_consensus.cc:504] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.379501 31400 leader_election.cc:288] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275), cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503)
I20260810 18:01:16.379930 30957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df029cff1fb442c88881304946f5e64" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" is_pre_election: true
I20260810 18:01:16.380086 30957 raft_consensus.cc:2454] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 0.
I20260810 18:01:16.380174 31249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df029cff1fb442c88881304946f5e64" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790" is_pre_election: true
I20260810 18:01:16.380375 31249 raft_consensus.cc:2454] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 0.
I20260810 18:01:16.380523 31032 leader_election.cc:302] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:16.380718 31400 raft_consensus.cc:2781] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:16.380767 31400 raft_consensus.cc:482] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:16.380802 31400 raft_consensus.cc:3037] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:16.382205 31400 raft_consensus.cc:504] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.382584 31400 leader_election.cc:288] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 election: Requested vote from peers 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275), cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503)
I20260810 18:01:16.383044 31249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df029cff1fb442c88881304946f5e64" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790"
I20260810 18:01:16.383052 30957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df029cff1fb442c88881304946f5e64" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005"
I20260810 18:01:16.383167 30957 raft_consensus.cc:3037] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:16.383193 31249 raft_consensus.cc:3037] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:16.384382 31249 raft_consensus.cc:2454] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 1.
I20260810 18:01:16.384397 30957 raft_consensus.cc:2454] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 1.
I20260810 18:01:16.384716 31032 leader_election.cc:302] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:16.384984 31400 raft_consensus.cc:2781] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:16.385082 31400 raft_consensus.cc:686] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 1 LEADER]: Becoming Leader. State: Replica: d26a8527c619486ca2f15ef8189e210a, State: Running, Role: LEADER
I20260810 18:01:16.385239 31400 consensus_queue.cc:237] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.387171 30795 catalog_manager.cc:5495] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a reported cstate change: term changed from 0 to 1, leader changed from <none> to d26a8527c619486ca2f15ef8189e210a (127.28.83.66). New cstate: current_term: 1 leader_uuid: "d26a8527c619486ca2f15ef8189e210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } health_report { overall_health: UNKNOWN } } }
I20260810 18:01:16.408684 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:16.404847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.28.83.66:34447)
0810 18:01:16.405240 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:16.405275 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:16.405288 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:01:16.405329 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:16.405331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:16.405373 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:16.405508 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:16.405729 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:16.406285 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:16.407518 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:16.407846 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:16.407953 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:16.408266 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:16.408340 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:16.408396 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:16.408510 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 18:01:16.408559 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":195,"threads_started":1}
I20260810 18:01:16.415592 30957 raft_consensus.cc:1261] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Refusing update from remote peer d26a8527c619486ca2f15ef8189e210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:16.415824 31250 raft_consensus.cc:1261] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Refusing update from remote peer d26a8527c619486ca2f15ef8189e210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:16.416097 31374 consensus_queue.cc:1035] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:16.416251 31400 consensus_queue.cc:1035] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:16.448987 30793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:16.449150 30793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:16.451239 30793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989: Sending DeleteTablet for 3 replicas of tablet 8df029cff1fb442c88881304946f5e64
I20260810 18:01:16.451843 30936 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df029cff1fb442c88881304946f5e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:16 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:40476
I20260810 18:01:16.451939 31083 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df029cff1fb442c88881304946f5e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:16 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:44478
I20260810 18:01:16.452070 31229 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df029cff1fb442c88881304946f5e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:16 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:39506
I20260810 18:01:16.452152 31408 tablet_replica.cc:330] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: stopping tablet replica
I20260810 18:01:16.452191 31407 tablet_replica.cc:330] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: stopping tablet replica
I20260810 18:01:16.452277 31408 raft_consensus.cc:2229] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:16.452308 31407 raft_consensus.cc:2229] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:16.452432 31407 raft_consensus.cc:2258] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:16.452468 31408 raft_consensus.cc:2258] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:16.452652 31406 tablet_replica.cc:330] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: stopping tablet replica
I20260810 18:01:16.452626 30793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:01:16.452786 31406 raft_consensus.cc:2229] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:16.452912 31406 raft_consensus.cc:2258] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 18:01:16.453070 30793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:16.453260 31408 ts_tablet_manager.cc:1905] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:16.453264 31407 ts_tablet_manager.cc:1905] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:16.453918 31406 ts_tablet_manager.cc:1905] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:16.455821 31407 ts_tablet_manager.cc:1918] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:16.456000 31407 log.cc:1196] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/8df029cff1fb442c88881304946f5e64
I20260810 18:01:16.456154 31408 ts_tablet_manager.cc:1918] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:16.456238 31408 log.cc:1196] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/8df029cff1fb442c88881304946f5e64
I20260810 18:01:16.456516 31407 ts_tablet_manager.cc:1939] T 8df029cff1fb442c88881304946f5e64 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Deleting consensus metadata
I20260810 18:01:16.456523 31408 ts_tablet_manager.cc:1939] T 8df029cff1fb442c88881304946f5e64 P d26a8527c619486ca2f15ef8189e210a: Deleting consensus metadata
I20260810 18:01:16.456971 31406 ts_tablet_manager.cc:1918] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:16.457060 31406 log.cc:1196] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/8df029cff1fb442c88881304946f5e64
I20260810 18:01:16.457336 31406 ts_tablet_manager.cc:1939] T 8df029cff1fb442c88881304946f5e64 P cf2dc2e3ccb046308e78478bba960790: Deleting consensus metadata
I20260810 18:01:16.457366 30779 catalog_manager.cc:4841] TS 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275): tablet 8df029cff1fb442c88881304946f5e64 (table test-table [id=8bf573f108374e7181d7bbf69e76064a]) successfully deleted
I20260810 18:01:16.457556 30779 catalog_manager.cc:4841] TS d26a8527c619486ca2f15ef8189e210a (127.28.83.66:34447): tablet 8df029cff1fb442c88881304946f5e64 (table test-table [id=8bf573f108374e7181d7bbf69e76064a]) successfully deleted
I20260810 18:01:16.458124 30781 catalog_manager.cc:4841] TS cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503): tablet 8df029cff1fb442c88881304946f5e64 (table test-table [id=8bf573f108374e7181d7bbf69e76064a]) successfully deleted
I20260810 18:01:16.460114 30936 tablet_service.cc:1467] Processing CreateTablet for tablet 305ad0cd0be04c819d9b376e17fd1976 (DEFAULT_TABLE table=test-table [id=a0d8152ecf4443ae902271188ac1291a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:16.460183 31083 tablet_service.cc:1467] Processing CreateTablet for tablet 305ad0cd0be04c819d9b376e17fd1976 (DEFAULT_TABLE table=test-table [id=a0d8152ecf4443ae902271188ac1291a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:16.460343 31229 tablet_service.cc:1467] Processing CreateTablet for tablet 305ad0cd0be04c819d9b376e17fd1976 (DEFAULT_TABLE table=test-table [id=a0d8152ecf4443ae902271188ac1291a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:16.460404 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 305ad0cd0be04c819d9b376e17fd1976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:16.460403 30936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 305ad0cd0be04c819d9b376e17fd1976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:16.460615 31229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 305ad0cd0be04c819d9b376e17fd1976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:16.462795 31418 tablet_bootstrap.cc:492] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Bootstrap starting.
I20260810 18:01:16.462831 31417 tablet_bootstrap.cc:492] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: Bootstrap starting.
I20260810 18:01:16.462822 31419 tablet_bootstrap.cc:492] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: Bootstrap starting.
I20260810 18:01:16.463620 31418 tablet_bootstrap.cc:654] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:16.464114 31417 tablet_bootstrap.cc:654] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:16.464809 31419 tablet_bootstrap.cc:654] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:16.465173 31418 tablet_bootstrap.cc:492] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005: No bootstrap required, opened a new log
I20260810 18:01:16.465258 31418 ts_tablet_manager.cc:1397] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 18:01:16.465723 31418 raft_consensus.cc:348] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.465843 31418 raft_consensus.cc:374] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:16.465878 31418 raft_consensus.cc:729] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd0d725e3d24fe7a7c3d278d5bf5005, State: Initialized, Role: FOLLOWER
I20260810 18:01:16.465884 31417 tablet_bootstrap.cc:492] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: No bootstrap required, opened a new log
I20260810 18:01:16.465965 31417 ts_tablet_manager.cc:1397] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:16.466015 31418 consensus_queue.cc:260] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.466279 31418 ts_tablet_manager.cc:1428] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:16.466369 31417 raft_consensus.cc:348] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.466499 31417 raft_consensus.cc:374] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:16.466532 31417 raft_consensus.cc:729] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26a8527c619486ca2f15ef8189e210a, State: Initialized, Role: FOLLOWER
I20260810 18:01:16.466647 31417 consensus_queue.cc:260] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.466748 31419 tablet_bootstrap.cc:492] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: No bootstrap required, opened a new log
I20260810 18:01:16.466856 31419 ts_tablet_manager.cc:1397] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260810 18:01:16.466952 31417 ts_tablet_manager.cc:1428] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:16.468134 31419 raft_consensus.cc:348] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.468293 31419 raft_consensus.cc:374] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:16.468344 31419 raft_consensus.cc:729] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2dc2e3ccb046308e78478bba960790, State: Initialized, Role: FOLLOWER
I20260810 18:01:16.468483 31419 consensus_queue.cc:260] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.468958 31419 ts_tablet_manager.cc:1428] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:16.619989 31374 raft_consensus.cc:482] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:16.620136 31374 raft_consensus.cc:504] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.620548 31374 leader_election.cc:288] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275), cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503)
I20260810 18:01:16.621014 30957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305ad0cd0be04c819d9b376e17fd1976" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" is_pre_election: true
I20260810 18:01:16.621081 31250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305ad0cd0be04c819d9b376e17fd1976" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790" is_pre_election: true
I20260810 18:01:16.621227 30957 raft_consensus.cc:2454] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 0.
I20260810 18:01:16.621268 31250 raft_consensus.cc:2454] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 0.
I20260810 18:01:16.621584 31032 leader_election.cc:302] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:16.621865 31374 raft_consensus.cc:2781] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:16.621932 31374 raft_consensus.cc:482] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:16.621984 31374 raft_consensus.cc:3037] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:16.623380 31374 raft_consensus.cc:504] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.623739 31374 leader_election.cc:288] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 election: Requested vote from peers 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275), cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503)
I20260810 18:01:16.624141 31250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305ad0cd0be04c819d9b376e17fd1976" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2dc2e3ccb046308e78478bba960790"
I20260810 18:01:16.624162 30957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "305ad0cd0be04c819d9b376e17fd1976" candidate_uuid: "d26a8527c619486ca2f15ef8189e210a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005"
I20260810 18:01:16.624251 31250 raft_consensus.cc:3037] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:16.624305 30957 raft_consensus.cc:3037] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:16.625325 30957 raft_consensus.cc:2454] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 1.
I20260810 18:01:16.625541 31250 raft_consensus.cc:2454] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26a8527c619486ca2f15ef8189e210a in term 1.
I20260810 18:01:16.625725 31032 leader_election.cc:302] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd0d725e3d24fe7a7c3d278d5bf5005, d26a8527c619486ca2f15ef8189e210a; no voters:
I20260810 18:01:16.625932 31374 raft_consensus.cc:2781] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:16.626006 31374 raft_consensus.cc:686] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 1 LEADER]: Becoming Leader. State: Replica: d26a8527c619486ca2f15ef8189e210a, State: Running, Role: LEADER
I20260810 18:01:16.626166 31374 consensus_queue.cc:237] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } }
I20260810 18:01:16.627796 30795 catalog_manager.cc:5495] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a reported cstate change: term changed from 0 to 1, leader changed from <none> to d26a8527c619486ca2f15ef8189e210a (127.28.83.66). New cstate: current_term: 1 leader_uuid: "d26a8527c619486ca2f15ef8189e210a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26a8527c619486ca2f15ef8189e210a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 } health_report { overall_health: UNKNOWN } } }
I20260810 18:01:16.663983 31249 raft_consensus.cc:1261] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Refusing update from remote peer d26a8527c619486ca2f15ef8189e210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:16.664008 30957 raft_consensus.cc:1261] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Refusing update from remote peer d26a8527c619486ca2f15ef8189e210a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:16.664472 31374 consensus_queue.cc:1035] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd0d725e3d24fe7a7c3d278d5bf5005" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:16.664649 31397 consensus_queue.cc:1035] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2dc2e3ccb046308e78478bba960790" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:01:16.679359 31388 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 18:01:18.228785 30794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:18.228930 30794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:18.231153 30794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 698c0532b4ce4a26ba80fb38a08be989: Sending DeleteTablet for 3 replicas of tablet 305ad0cd0be04c819d9b376e17fd1976
I20260810 18:01:18.231659 30936 tablet_service.cc:1514] Processing DeleteTablet for tablet 305ad0cd0be04c819d9b376e17fd1976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:18 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:40476
I20260810 18:01:18.231786 31083 tablet_service.cc:1514] Processing DeleteTablet for tablet 305ad0cd0be04c819d9b376e17fd1976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:18 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:44478
I20260810 18:01:18.232005 31458 tablet_replica.cc:330] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005: stopping tablet replica
I20260810 18:01:18.232131 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 30871
I20260810 18:01:18.232123 31229 tablet_service.cc:1514] Processing DeleteTablet for tablet 305ad0cd0be04c819d9b376e17fd1976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:18 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:39506
I20260810 18:01:18.232154 31458 raft_consensus.cc:2229] T 305ad0cd0be04c819d9b376e17fd1976 P 5dd0d725e3d24fe7a7c3d278d5bf5005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:18.232175 31459 tablet_replica.cc:330] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: stopping tablet replica
I20260810 18:01:18.233923 31459 raft_consensus.cc:2229] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:18.234139 31459 raft_consensus.cc:2258] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:18.234603 31460 tablet_replica.cc:330] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: stopping tablet replica
I20260810 18:01:18.234772 31460 raft_consensus.cc:2229] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:18.234911 31460 raft_consensus.cc:2258] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:18.235047 31459 ts_tablet_manager.cc:1905] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:18.235549 31460 ts_tablet_manager.cc:1905] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:18.237902 31460 ts_tablet_manager.cc:1918] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:01:18.237985 31460 log.cc:1196] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/305ad0cd0be04c819d9b376e17fd1976
I20260810 18:01:18.238267 31460 ts_tablet_manager.cc:1939] T 305ad0cd0be04c819d9b376e17fd1976 P cf2dc2e3ccb046308e78478bba960790: Deleting consensus metadata
I20260810 18:01:18.238597 31459 ts_tablet_manager.cc:1918] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:01:18.238687 31459 log.cc:1196] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/305ad0cd0be04c819d9b376e17fd1976
I20260810 18:01:18.238998 31459 ts_tablet_manager.cc:1939] T 305ad0cd0be04c819d9b376e17fd1976 P d26a8527c619486ca2f15ef8189e210a: Deleting consensus metadata
I20260810 18:01:18.239002 30781 catalog_manager.cc:4841] TS cf2dc2e3ccb046308e78478bba960790 (127.28.83.67:41503): tablet 305ad0cd0be04c819d9b376e17fd1976 (table test-table [id=a0d8152ecf4443ae902271188ac1291a]) successfully deleted
I20260810 18:01:18.239740 30779 catalog_manager.cc:4841] TS d26a8527c619486ca2f15ef8189e210a (127.28.83.66:34447): tablet 305ad0cd0be04c819d9b376e17fd1976 (table test-table [id=a0d8152ecf4443ae902271188ac1291a]) successfully deleted
W20260810 18:01:18.242157 30779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:41275 (error 108)
I20260810 18:01:18.243045 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 31015
I20260810 18:01:18.243507 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:18.243018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:41275 (local address 127.0.0.1:59318)
0810 18:01:18.243304 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:18.243385 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
W20260810 18:01:18.243969 30779 catalog_manager.cc:4584] TS 5dd0d725e3d24fe7a7c3d278d5bf5005 (127.28.83.65:41275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 305ad0cd0be04c819d9b376e17fd1976: Network error: Client connection negotiation failed: client connection to 127.28.83.65:41275: connect: Connection refused (error 111)
I20260810 18:01:18.252038 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 31162
I20260810 18:01:18.258695 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 30762
2026-08-10T18:01:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6535 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:18 dist-test-slave-srgv krb5kdc[31465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:18 dist-test-slave-srgv krb5kdc[31465](info): set up 2 sockets
Aug 10 18:01:18 dist-test-slave-srgv krb5kdc[31465](info): commencing operation
krb5kdc: starting...
W20260810 18:01:19.339881 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:19 dist-test-slave-srgv krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:19.468597 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36467
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:36467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:19.579885 31480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:19.580247 31480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:19.580325 31480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:19.584692 31480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:19.584800 31480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:19.584836 31480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:19.584877 31480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:19.584910 31480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:19.590485 31480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:36467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:19.591688 31480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:19.592738 31480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:19.601257 31485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:19.601332 31488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:19.601610 31486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:19.601540 31480 server_base.cc:1034] running on GCE node
I20260810 18:01:19.602273 31480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:19.603403 31480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:19.604612 31480 hybrid_clock.cc:648] HybridClock initialized: now 1786384879604596 us; error 31 us; skew 500 ppm
Aug 10 18:01:19 dist-test-slave-srgv krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384879, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:19.607456 31480 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:19.608467 31480 webserver.cc:460] Webserver started at http://127.28.83.126:45887/ using document root <none> and password file <none>
I20260810 18:01:19.609051 31480 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:19.609148 31480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:19.609392 31480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:19.611510 31480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "5c577d1920e14393a17789b71659e89e"
format_stamp: "Formatted at 2026-08-10 18:01:19 on dist-test-slave-srgv"
server_key: "fefe5e35f01cb818a685de6aa469f0c4"
server_key_iv: "da6e469416da73db1fdd19f4655615a3"
server_key_version: "encryptionkey@0"
I20260810 18:01:19.612111 31480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "5c577d1920e14393a17789b71659e89e"
format_stamp: "Formatted at 2026-08-10 18:01:19 on dist-test-slave-srgv"
server_key: "fefe5e35f01cb818a685de6aa469f0c4"
server_key_iv: "da6e469416da73db1fdd19f4655615a3"
server_key_version: "encryptionkey@0"
I20260810 18:01:19.615998 31480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:01:19.618444 31495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:19.619681 31480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:19.619848 31480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "5c577d1920e14393a17789b71659e89e"
format_stamp: "Formatted at 2026-08-10 18:01:19 on dist-test-slave-srgv"
server_key: "fefe5e35f01cb818a685de6aa469f0c4"
server_key_iv: "da6e469416da73db1fdd19f4655615a3"
server_key_version: "encryptionkey@0"
I20260810 18:01:19.619968 31480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:19.635614 31480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:19.639662 31480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:19.639889 31480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:19.648262 31480 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:36467
I20260810 18:01:19.648324 31551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:36467 every 8 connection(s)
I20260810 18:01:19.649494 31480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:19.652315 31552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:19.655861 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 31480
I20260810 18:01:19.655992 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:19.656311 29005 external_mini_cluster.cc:1428] Setting key d4d4741fda3692328caff4408e43daee
I20260810 18:01:19.658563 31552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e: Bootstrap starting.
I20260810 18:01:19.661135 31552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:19.662127 31552 log.cc:824] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:19.664412 31552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e: No bootstrap required, opened a new log
Aug 10 18:01:19 dist-test-slave-srgv krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:19.668313 31552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c577d1920e14393a17789b71659e89e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36467 } }
I20260810 18:01:19.668583 31552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:19.668665 31552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c577d1920e14393a17789b71659e89e, State: Initialized, Role: FOLLOWER
I20260810 18:01:19.669122 31552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c577d1920e14393a17789b71659e89e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36467 } }
I20260810 18:01:19.669296 31552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:19.669377 31552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:19.669519 31552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:19.670760 31552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c577d1920e14393a17789b71659e89e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36467 } }
I20260810 18:01:19.671108 31552 leader_election.cc:302] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c577d1920e14393a17789b71659e89e; no voters:
I20260810 18:01:19.671423 31552 leader_election.cc:288] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:19.671566 31557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:19.671732 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:19.657843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.28.83.126:36467)
0810 18:01:19.658282 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:19.658299 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:19.658336 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:01:19.659032 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:19.659051 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:19.659390 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:19.659693 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:19.660597 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:19.661737 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:19.662862 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:19.663355 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:19.663476 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:19.666449 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:19.666454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:19.666468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:19.666496 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:19.668339 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:19.668880 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:19.668885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:19.668890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:19.669062 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:19.669401 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:19.669405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:19.669406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:19.669919 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:19.670090 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:19.670422 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 18:01:19.670745 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":136,"threads_started":1}
I20260810 18:01:19.671793 31557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [term 1 LEADER]: Becoming Leader. State: Replica: 5c577d1920e14393a17789b71659e89e, State: Running, Role: LEADER
I20260810 18:01:19.672423 31557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c577d1920e14393a17789b71659e89e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36467 } }
I20260810 18:01:19.672775 31552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:19.674062 31559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c577d1920e14393a17789b71659e89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c577d1920e14393a17789b71659e89e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36467 } } }
I20260810 18:01:19.674183 31559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:19.674510 31558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c577d1920e14393a17789b71659e89e. Latest consensus state: current_term: 1 leader_uuid: "5c577d1920e14393a17789b71659e89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c577d1920e14393a17789b71659e89e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36467 } } }
I20260810 18:01:19.674589 31558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e [sys.catalog]: This master's current role is: LEADER
W20260810 18:01:19.677953 31572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:01:19.678035 31572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:01:19.678128 31566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:19.678814 31566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:19.683972 31566 catalog_manager.cc:1347] Generated new cluster ID: d5a97f9a0eba4f79be0b4eda356bf689
I20260810 18:01:19.684055 31566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:19.704484 31566 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:19.705485 31566 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:19.711630 31566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c577d1920e14393a17789b71659e89e: Generated new TSK 0
I20260810 18:01:19.712327 31566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:19.742681 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36467
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:19.858539 31578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:19.858901 31578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:19.858960 31578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:19.863158 31578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:19.863224 31578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:19.863283 31578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:19.868706 31578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:19.869935 31578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:19.870981 31578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:19.881206 31584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:19.881209 31583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:19.881340 31586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:19.881421 31578 server_base.cc:1034] running on GCE node
I20260810 18:01:19.881860 31578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:19.882496 31578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:19.883688 31578 hybrid_clock.cc:648] HybridClock initialized: now 1786384879883652 us; error 47 us; skew 500 ppm
Aug 10 18:01:19 dist-test-slave-srgv krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384879, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:19.886507 31578 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:19.887388 31578 webserver.cc:460] Webserver started at http://127.28.83.65:39183/ using document root <none> and password file <none>
I20260810 18:01:19.887895 31578 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:19.887954 31578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:19.888116 31578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:19.890177 31578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "848d7338bc3840afac542a0b19574018"
format_stamp: "Formatted at 2026-08-10 18:01:19 on dist-test-slave-srgv"
server_key: "0f1119f43592357ac2700c7362374363"
server_key_iv: "3f1b97225dcda0a24366852117f52216"
server_key_version: "encryptionkey@0"
I20260810 18:01:19.890700 31578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "848d7338bc3840afac542a0b19574018"
format_stamp: "Formatted at 2026-08-10 18:01:19 on dist-test-slave-srgv"
server_key: "0f1119f43592357ac2700c7362374363"
server_key_iv: "3f1b97225dcda0a24366852117f52216"
server_key_version: "encryptionkey@0"
I20260810 18:01:19.894279 31578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:01:19.896517 31593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:19.897758 31578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:19.897850 31578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "848d7338bc3840afac542a0b19574018"
format_stamp: "Formatted at 2026-08-10 18:01:19 on dist-test-slave-srgv"
server_key: "0f1119f43592357ac2700c7362374363"
server_key_iv: "3f1b97225dcda0a24366852117f52216"
server_key_version: "encryptionkey@0"
I20260810 18:01:19.897933 31578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:19.913311 31578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:19.916296 31578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:19.916455 31578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:19.916982 31578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:19.917866 31578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:19.917917 31578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:19.917956 31578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:19.917971 31578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:19.927376 31578 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:44541
I20260810 18:01:19.927412 31711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:44541 every 8 connection(s)
I20260810 18:01:19.928305 31578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:19.930154 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 31578
I20260810 18:01:19.930318 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:19.930727 29005 external_mini_cluster.cc:1428] Setting key 253b33de1fb81f50e85a2659481d6949
Aug 10 18:01:19 dist-test-slave-srgv krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384879, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:01:19.943837 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:19.930085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:42571 (local address 127.28.83.126:36467)
0810 18:01:19.930260 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:19.930265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:19.931161 (+ 896us) server_negotiation.cc:407] Connection header received
0810 18:01:19.932455 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:19.932458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:19.932511 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:19.932608 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:19.933860 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:19.934552 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:19.936356 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:19.936593 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:19.936676 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:19.939785 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:19.939789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:19.939791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:19.939815 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:19.941519 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:19.942067 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:19.942070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:19.942072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:19.942160 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:19.942561 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:19.942567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:19.942572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:19.942771 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:19.942879 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:19.943530 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 18:01:19.943667 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:19.945056 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:19.930484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36467 (local address 127.28.83.65:42571)
0810 18:01:19.930976 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:19.931029 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:19.932249 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:19.932773 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:19.932780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:19.933135 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:19.933705 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:19.933719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:19.934689 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:19.934695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:19.936220 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:19.936235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:19.936769 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:19.936776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:19.937017 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:19.937765 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:19.937792 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:19.939589 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:19.941646 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:19.941653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:19.941670 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:19.941946 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:19.942260 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:19.942264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:19.942266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:19.942457 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:19.942894 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:19.942902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:19.943179 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 18:01:19.943473 (+ 294us) client_negotiation.cc:240] Negotiation successful
0810 18:01:19.943892 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":141,"threads_started":1}
I20260810 18:01:19.946900 31712 heartbeater.cc:347] Connected to a master server at 127.28.83.126:36467
I20260810 18:01:19.947232 31712 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:19.947996 31712 heartbeater.cc:511] Master 127.28.83.126:36467 requested a full tablet report, sending...
I20260810 18:01:19.950723 31513 ts_manager.cc:194] Registered new tserver with Master: 848d7338bc3840afac542a0b19574018 (127.28.83.65:44541)
I20260810 18:01:19.952280 31513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:42571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:19.964913 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36467
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:20.078608 31719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:20.078902 31719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:20.078948 31719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:20.083170 31719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:20.083242 31719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:20.083305 31719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:20.089088 31719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:20.090257 31719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:20.091287 31719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:01:20.100986 31719 server_base.cc:1034] running on GCE node
W20260810 18:01:20.100978 31727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:20.100950 31724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:20.100930 31725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:20.101723 31719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:20.102404 31719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:20.103587 31719 hybrid_clock.cc:648] HybridClock initialized: now 1786384880103562 us; error 44 us; skew 500 ppm
Aug 10 18:01:20 dist-test-slave-srgv krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384880, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:20.106459 31719 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:20.107497 31719 webserver.cc:460] Webserver started at http://127.28.83.66:46129/ using document root <none> and password file <none>
I20260810 18:01:20.108048 31719 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:20.108142 31719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:20.108382 31719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:20.110463 31719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "ba5cdbb9e04f4dbcaa9164720a9aedc0"
format_stamp: "Formatted at 2026-08-10 18:01:20 on dist-test-slave-srgv"
server_key: "e437f5c000cc2d4ebcf650cd747885e5"
server_key_iv: "4aa3b3c4bf8a35e63221c2e25ae42b1b"
server_key_version: "encryptionkey@0"
I20260810 18:01:20.111049 31719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "ba5cdbb9e04f4dbcaa9164720a9aedc0"
format_stamp: "Formatted at 2026-08-10 18:01:20 on dist-test-slave-srgv"
server_key: "e437f5c000cc2d4ebcf650cd747885e5"
server_key_iv: "4aa3b3c4bf8a35e63221c2e25ae42b1b"
server_key_version: "encryptionkey@0"
I20260810 18:01:20.114753 31719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:01:20.117057 31734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:20.118347 31719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:20.118484 31719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "ba5cdbb9e04f4dbcaa9164720a9aedc0"
format_stamp: "Formatted at 2026-08-10 18:01:20 on dist-test-slave-srgv"
server_key: "e437f5c000cc2d4ebcf650cd747885e5"
server_key_iv: "4aa3b3c4bf8a35e63221c2e25ae42b1b"
server_key_version: "encryptionkey@0"
I20260810 18:01:20.118597 31719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:20.134485 31719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:20.138084 31719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:20.138314 31719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:20.138954 31719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:20.139817 31719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:20.139890 31719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:20.139963 31719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:20.140004 31719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:20.149413 31719 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:39973
I20260810 18:01:20.149457 31852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:39973 every 8 connection(s)
I20260810 18:01:20.150374 31719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:20.152129 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 31719
I20260810 18:01:20.152251 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:20.152563 29005 external_mini_cluster.cc:1428] Setting key ce1ddfea2ae6076496dc7ae75e52afcf
Aug 10 18:01:20 dist-test-slave-srgv krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384880, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:01:20.165271 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:20.152304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:60989 (local address 127.28.83.126:36467)
0810 18:01:20.152483 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:20.152488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:20.153452 (+ 964us) server_negotiation.cc:407] Connection header received
0810 18:01:20.154920 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:20.154923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:20.154975 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:20.155080 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:20.156249 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:20.156767 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:20.157961 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:20.158190 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:20.158310 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:20.161231 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:20.161233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:20.161236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:20.161261 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:20.163001 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:20.163572 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:20.163575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:20.163577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:20.163666 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:20.164016 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:20.164020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:20.164022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:20.164288 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:20.164402 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:20.164973 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 18:01:20.165107 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 18:01:20.166270 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:20.152738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36467 (local address 127.28.83.66:60989)
0810 18:01:20.153274 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:20.153335 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:20.154712 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:20.155255 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:20.155262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:20.155621 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:20.156102 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:20.156115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:20.156891 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:20.156895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:20.157838 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:20.157847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:20.158351 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:20.158355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:20.158537 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:20.159163 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:20.159180 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:20.161037 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:20.163132 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:20.163138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:20.163161 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:20.163453 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:20.163763 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:20.163766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:20.163768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:20.163912 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:20.164430 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:20.164439 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:20.164688 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 18:01:20.164930 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 18:01:20.165282 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
I20260810 18:01:20.167443 31853 heartbeater.cc:347] Connected to a master server at 127.28.83.126:36467
I20260810 18:01:20.167723 31853 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:20.168236 31853 heartbeater.cc:511] Master 127.28.83.126:36467 requested a full tablet report, sending...
I20260810 18:01:20.169345 31513 ts_manager.cc:194] Registered new tserver with Master: ba5cdbb9e04f4dbcaa9164720a9aedc0 (127.28.83.66:39973)
I20260810 18:01:20.169973 31513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:60989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:01:20.181058 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36467
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:20.296919 31858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:20.297236 31858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:20.297322 31858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:20.301558 31858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:20.301636 31858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:20.301729 31858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:20.307166 31858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:20.308254 31858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:20.309278 31858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:20.318830 31863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:20.318814 31864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:20.318873 31866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:20.319037 31858 server_base.cc:1034] running on GCE node
I20260810 18:01:20.319556 31858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:20.320235 31858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:20.321475 31858 hybrid_clock.cc:648] HybridClock initialized: now 1786384880321436 us; error 41 us; skew 500 ppm
Aug 10 18:01:20 dist-test-slave-srgv krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384880, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:20.324676 31858 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:20.325701 31858 webserver.cc:460] Webserver started at http://127.28.83.67:41895/ using document root <none> and password file <none>
I20260810 18:01:20.326259 31858 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:20.326350 31858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:20.326589 31858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:20.328697 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "2ab9d25910dd444290bdc020c85fb4fc"
format_stamp: "Formatted at 2026-08-10 18:01:20 on dist-test-slave-srgv"
server_key: "08ef93ef912672fa0716a8263029d49c"
server_key_iv: "55b773869768163be67b1ec1c0bedc7c"
server_key_version: "encryptionkey@0"
I20260810 18:01:20.329272 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "2ab9d25910dd444290bdc020c85fb4fc"
format_stamp: "Formatted at 2026-08-10 18:01:20 on dist-test-slave-srgv"
server_key: "08ef93ef912672fa0716a8263029d49c"
server_key_iv: "55b773869768163be67b1ec1c0bedc7c"
server_key_version: "encryptionkey@0"
I20260810 18:01:20.332948 31858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:01:20.335474 31873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:20.336661 31858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:20.336800 31858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "2ab9d25910dd444290bdc020c85fb4fc"
format_stamp: "Formatted at 2026-08-10 18:01:20 on dist-test-slave-srgv"
server_key: "08ef93ef912672fa0716a8263029d49c"
server_key_iv: "55b773869768163be67b1ec1c0bedc7c"
server_key_version: "encryptionkey@0"
I20260810 18:01:20.336916 31858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:20.354390 31858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:20.357991 31858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:20.358219 31858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:20.358850 31858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:20.359697 31858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:20.359769 31858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:20.359844 31858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:20.359889 31858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:20.369589 31858 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:39977
I20260810 18:01:20.369603 31992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:39977 every 8 connection(s)
I20260810 18:01:20.370543 31858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:20.378588 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 31858
I20260810 18:01:20.378724 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:20.379004 29005 external_mini_cluster.cc:1428] Setting key 22c5b9c5bb0c58d02d3c820c1a03feb6
Aug 10 18:01:20 dist-test-slave-srgv krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384880, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:20.385150 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:20.372483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:41943 (local address 127.28.83.126:36467)
0810 18:01:20.372673 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:20.372677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:20.373592 (+ 915us) server_negotiation.cc:407] Connection header received
0810 18:01:20.375040 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:20.375043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:20.375105 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:20.375213 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:20.376396 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:20.376922 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:20.378134 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:20.378359 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:20.378472 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:20.381446 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:20.381453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:20.381459 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:20.381493 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:20.382989 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:20.383558 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:20.383564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:20.383568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:20.383650 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:20.384004 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:20.384008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:20.384010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:20.384215 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:20.384328 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:20.384846 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 18:01:20.384973 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 18:01:20.386070 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:20.372923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36467 (local address 127.28.83.67:41943)
0810 18:01:20.373378 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:20.373478 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:20.374824 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:20.375387 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:20.375393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:20.375748 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:20.376248 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:20.376262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:20.377061 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:20.377069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:20.377996 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:20.378007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:20.378517 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:20.378521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:20.378741 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:20.379444 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:20.379473 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:20.381229 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:20.383125 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:20.383132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:20.383147 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:20.383450 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:20.383745 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:20.383749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:20.383751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:20.383904 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:20.384327 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:20.384333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:20.384607 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 18:01:20.384805 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 18:01:20.385159 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":127,"threads_started":1}
I20260810 18:01:20.387351 31993 heartbeater.cc:347] Connected to a master server at 127.28.83.126:36467
I20260810 18:01:20.387637 31993 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:20.388177 31993 heartbeater.cc:511] Master 127.28.83.126:36467 requested a full tablet report, sending...
I20260810 18:01:20.389243 31513 ts_manager.cc:194] Registered new tserver with Master: 2ab9d25910dd444290bdc020c85fb4fc (127.28.83.67:39977)
I20260810 18:01:20.389899 31513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:41943
I20260810 18:01:20.393244 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:01:20.407388 31555 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:20.405734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.28.83.126:36467)
0810 18:01:20.405903 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:20.405908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:20.406070 (+ 162us) server_negotiation.cc:407] Connection header received
0810 18:01:20.406279 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:20.406282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:20.406334 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:20.406426 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:20.407222 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57446: BlockingRecv error: recv got EOF from 127.0.0.1:57446 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:01:20.408833 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 31578
I20260810 18:01:20.415360 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 31719
I20260810 18:01:20.421936 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 31858
I20260810 18:01:20.429325 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 31480
2026-08-10T18:01:20Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2168 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:20 dist-test-slave-srgv krb5kdc[32005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:20 dist-test-slave-srgv krb5kdc[32005](info): set up 2 sockets
Aug 10 18:01:20 dist-test-slave-srgv krb5kdc[32005](info): commencing operation
krb5kdc: starting...
W20260810 18:01:21.484320 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:21 dist-test-slave-srgv krb5kdc[32005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:21.612027 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36625
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:36625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:21.725271 32018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:21.725632 32018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:21.725677 32018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:21.729553 32018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:21.729602 32018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:21.729621 32018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:21.729637 32018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:21.729652 32018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:21.734826 32018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:36625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:21.735874 32018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:21.736843 32018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:21.745345 32029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:21.745345 32024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:21.745361 32025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:21.745658 32018 server_base.cc:1034] running on GCE node
I20260810 18:01:21.746353 32018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:21.747555 32018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:21.748781 32018 hybrid_clock.cc:648] HybridClock initialized: now 1786384881748755 us; error 44 us; skew 500 ppm
Aug 10 18:01:21 dist-test-slave-srgv krb5kdc[32005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384881, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:21.751680 32018 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:21.752661 32018 webserver.cc:460] Webserver started at http://127.28.83.126:37339/ using document root <none> and password file <none>
I20260810 18:01:21.753182 32018 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:21.753252 32018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:21.753486 32018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:21.755506 32018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "efaba2f64c25463fba119947025b6e1f"
format_stamp: "Formatted at 2026-08-10 18:01:21 on dist-test-slave-srgv"
server_key: "fa8072afc95c7fba8c1520e728cf01f2"
server_key_iv: "79969b873a52398a3973cb39d2dab244"
server_key_version: "encryptionkey@0"
I20260810 18:01:21.756033 32018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "efaba2f64c25463fba119947025b6e1f"
format_stamp: "Formatted at 2026-08-10 18:01:21 on dist-test-slave-srgv"
server_key: "fa8072afc95c7fba8c1520e728cf01f2"
server_key_iv: "79969b873a52398a3973cb39d2dab244"
server_key_version: "encryptionkey@0"
I20260810 18:01:21.759776 32018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260810 18:01:21.762157 32036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:21.763448 32018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 18:01:21.763593 32018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "efaba2f64c25463fba119947025b6e1f"
format_stamp: "Formatted at 2026-08-10 18:01:21 on dist-test-slave-srgv"
server_key: "fa8072afc95c7fba8c1520e728cf01f2"
server_key_iv: "79969b873a52398a3973cb39d2dab244"
server_key_version: "encryptionkey@0"
I20260810 18:01:21.763710 32018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:21.780710 32018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:21.784324 32018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:21.784548 32018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:21.792147 32018 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:36625
I20260810 18:01:21.792546 32089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:36625 every 8 connection(s)
I20260810 18:01:21.793321 32018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:21.796155 32090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:21.798619 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 32018
I20260810 18:01:21.798730 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:21.799022 29005 external_mini_cluster.cc:1428] Setting key d0aa5885e3765590a63f0acd02e52bd8
I20260810 18:01:21.802174 32090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f: Bootstrap starting.
I20260810 18:01:21.804394 32090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:21.805130 32090 log.cc:824] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:21.807168 32090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f: No bootstrap required, opened a new log
Aug 10 18:01:21 dist-test-slave-srgv krb5kdc[32005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:21.810665 32090 raft_consensus.cc:348] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efaba2f64c25463fba119947025b6e1f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36625 } }
I20260810 18:01:21.810910 32090 raft_consensus.cc:374] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:21.810986 32090 raft_consensus.cc:729] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efaba2f64c25463fba119947025b6e1f, State: Initialized, Role: FOLLOWER
I20260810 18:01:21.811496 32090 consensus_queue.cc:260] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efaba2f64c25463fba119947025b6e1f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36625 } }
I20260810 18:01:21.811663 32090 raft_consensus.cc:388] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:21.811733 32090 raft_consensus.cc:482] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:21.811825 32090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:21.813054 32090 raft_consensus.cc:504] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efaba2f64c25463fba119947025b6e1f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36625 } }
I20260810 18:01:21.813437 32090 leader_election.cc:302] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efaba2f64c25463fba119947025b6e1f; no voters:
I20260810 18:01:21.813774 32090 leader_election.cc:288] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:21.813890 32096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:21.814168 32096 raft_consensus.cc:686] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [term 1 LEADER]: Becoming Leader. State: Replica: efaba2f64c25463fba119947025b6e1f, State: Running, Role: LEADER
I20260810 18:01:21.814203 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:21.800399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.28.83.126:36625)
0810 18:01:21.800848 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:21.800863 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:21.800898 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:01:21.801609 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:21.801629 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:21.801957 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:21.802297 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:21.803146 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:21.804130 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:21.805082 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:21.806018 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:21.806273 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:21.808954 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:21.808960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:21.808984 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:21.809011 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:21.810847 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:21.811373 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:21.811379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:21.811384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:21.811535 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:21.811872 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:21.811876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:21.811879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:21.812391 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:21.812586 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:21.812864 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 18:01:21.813173 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260810 18:01:21.814548 32096 consensus_queue.cc:237] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efaba2f64c25463fba119947025b6e1f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36625 } }
I20260810 18:01:21.815003 32090 sys_catalog.cc:564] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:21.816319 32097 sys_catalog.cc:455] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efaba2f64c25463fba119947025b6e1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efaba2f64c25463fba119947025b6e1f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36625 } } }
I20260810 18:01:21.816443 32097 sys_catalog.cc:458] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:21.816747 32098 sys_catalog.cc:455] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader efaba2f64c25463fba119947025b6e1f. Latest consensus state: current_term: 1 leader_uuid: "efaba2f64c25463fba119947025b6e1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efaba2f64c25463fba119947025b6e1f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36625 } } }
I20260810 18:01:21.816839 32098 sys_catalog.cc:458] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:21.817176 32105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:21.817767 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:01:21.820756 32105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:21.826898 32105 catalog_manager.cc:1347] Generated new cluster ID: b523f567ff5f455e918636a732eabe05
I20260810 18:01:21.826964 32105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:21.837306 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:21.831235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.28.83.126:36625)
0810 18:01:21.831378 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:21.831384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:21.831665 (+ 281us) server_negotiation.cc:407] Connection header received
0810 18:01:21.831793 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:21.831796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:21.831845 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:21.831928 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:21.832597 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:21.833457 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:21.834670 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:21.834999 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:21.835102 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:21.836177 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:21.836180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:21.836183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:21.836795 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:21.836827 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:21.836845 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:21.836961 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:21.837083 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 18:01:21.837154 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:01:21.837782 32105 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:21.838913 32105 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:21.845976 32105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efaba2f64c25463fba119947025b6e1f: Generated new TSK 0
I20260810 18:01:21.846802 32105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:01:21.856791 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 32018
2026-08-10T18:01:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1425 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:21 dist-test-slave-srgv krb5kdc[32125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:21 dist-test-slave-srgv krb5kdc[32125](info): set up 2 sockets
Aug 10 18:01:21 dist-test-slave-srgv krb5kdc[32125](info): commencing operation
krb5kdc: starting...
W20260810 18:01:22.913101 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:22 dist-test-slave-srgv krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:23.040189 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:35259
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:35259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:23.153143 32140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:23.153456 32140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:23.153498 32140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:23.157802 32140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:23.157869 32140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:23.157889 32140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:23.157907 32140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:23.157960 32140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:23.163316 32140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:35259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:23.164484 32140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:23.165562 32140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:23.173942 32148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:23.173949 32145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:23.173949 32146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:23.174352 32140 server_base.cc:1034] running on GCE node
I20260810 18:01:23.174928 32140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:23.176038 32140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:23.177258 32140 hybrid_clock.cc:648] HybridClock initialized: now 1786384883177230 us; error 47 us; skew 500 ppm
Aug 10 18:01:23 dist-test-slave-srgv krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:23.180136 32140 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:23.181072 32140 webserver.cc:460] Webserver started at http://127.28.83.126:44579/ using document root <none> and password file <none>
I20260810 18:01:23.181689 32140 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:23.181756 32140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:23.181928 32140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:23.183987 32140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "2b9410d19d754f5bb4cb151b3e5a811a"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "055bc45d5677870c6494b45dd7b4bc4b"
server_key_iv: "dfe6d38240daa7b601792fd44d57b67f"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.184557 32140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "2b9410d19d754f5bb4cb151b3e5a811a"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "055bc45d5677870c6494b45dd7b4bc4b"
server_key_iv: "dfe6d38240daa7b601792fd44d57b67f"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.188422 32140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:01:23.191294 32155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.192641 32140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:23.192799 32140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "2b9410d19d754f5bb4cb151b3e5a811a"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "055bc45d5677870c6494b45dd7b4bc4b"
server_key_iv: "dfe6d38240daa7b601792fd44d57b67f"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.192919 32140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:23.210089 32140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:23.213022 32140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:23.213241 32140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:23.220371 32140 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:35259
I20260810 18:01:23.220685 32208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:35259 every 8 connection(s)
I20260810 18:01:23.221350 32140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:23.224082 32209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:23.227284 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 32140
I20260810 18:01:23.227396 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:23.227721 29005 external_mini_cluster.cc:1428] Setting key 2f71ee777c5dad264ebe9e77fd9e9661
I20260810 18:01:23.230042 32209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a: Bootstrap starting.
I20260810 18:01:23.232191 32209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:23.232926 32209 log.cc:824] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:23.234863 32209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a: No bootstrap required, opened a new log
Aug 10 18:01:23 dist-test-slave-srgv krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:23.238435 32209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9410d19d754f5bb4cb151b3e5a811a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35259 } }
I20260810 18:01:23.238708 32209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:23.238791 32209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9410d19d754f5bb4cb151b3e5a811a, State: Initialized, Role: FOLLOWER
I20260810 18:01:23.239243 32209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9410d19d754f5bb4cb151b3e5a811a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35259 } }
I20260810 18:01:23.239398 32209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:23.239486 32209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:23.239609 32209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:23.240566 32209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9410d19d754f5bb4cb151b3e5a811a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35259 } }
I20260810 18:01:23.240914 32209 leader_election.cc:302] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b9410d19d754f5bb4cb151b3e5a811a; no voters:
I20260810 18:01:23.241302 32209 leader_election.cc:288] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:23.241433 32215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:23.241675 32215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [term 1 LEADER]: Becoming Leader. State: Replica: 2b9410d19d754f5bb4cb151b3e5a811a, State: Running, Role: LEADER
I20260810 18:01:23.242057 32215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9410d19d754f5bb4cb151b3e5a811a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35259 } }
I20260810 18:01:23.242483 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:23.229107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.28.83.126:35259)
0810 18:01:23.229609 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:23.229625 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:23.229660 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:01:23.230318 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:23.230335 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:23.230650 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:23.230988 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:23.231818 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.232863 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.233824 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.234290 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.234458 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:23.236992 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:23.236997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:23.237010 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:23.237037 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:23.239097 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:23.239601 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:23.239607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:23.239613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:23.239776 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:23.240076 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:23.240079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:23.240081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:23.240493 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:23.240713 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:23.241021 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 18:01:23.241378 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":127,"threads_started":1}
I20260810 18:01:23.242635 32209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:23.245532 32217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b9410d19d754f5bb4cb151b3e5a811a. Latest consensus state: current_term: 1 leader_uuid: "2b9410d19d754f5bb4cb151b3e5a811a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9410d19d754f5bb4cb151b3e5a811a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35259 } } }
I20260810 18:01:23.245616 32216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b9410d19d754f5bb4cb151b3e5a811a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9410d19d754f5bb4cb151b3e5a811a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35259 } } }
I20260810 18:01:23.245947 32217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:23.245947 32216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a [sys.catalog]: This master's current role is: LEADER
W20260810 18:01:23.247994 32230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:01:23.248092 32230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:01:23.248168 32231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:23.249022 32231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:23.255254 32231 catalog_manager.cc:1347] Generated new cluster ID: 6e71b82c52ce442fbf895c9592450486
I20260810 18:01:23.255371 32231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:23.269457 32231 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:23.270843 32231 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:23.278283 32231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b9410d19d754f5bb4cb151b3e5a811a: Generated new TSK 0
I20260810 18:01:23.279052 32231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:23.316471 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35259
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:23.434448 32236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:23.434796 32236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:23.434888 32236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:23.439146 32236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:23.439257 32236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:23.439357 32236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:23.445084 32236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:23.446307 32236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:23.447407 32236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:23.458004 32243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:23.458076 32242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:23.458004 32245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:23.458348 32236 server_base.cc:1034] running on GCE node
I20260810 18:01:23.458814 32236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:23.459527 32236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:23.460740 32236 hybrid_clock.cc:648] HybridClock initialized: now 1786384883460712 us; error 42 us; skew 500 ppm
Aug 10 18:01:23 dist-test-slave-srgv krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:23.468681 32236 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:23.469779 32236 webserver.cc:460] Webserver started at http://127.28.83.65:36075/ using document root <none> and password file <none>
I20260810 18:01:23.470396 32236 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:23.470499 32236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:23.470748 32236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:23.473093 32236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "d2a9687c5ef845919d9f39774b4305b6"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "6a40276161f0dc7b29121a2ab1254624"
server_key_iv: "93a8479ac1d34f33534f1abbf87fb25a"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.473953 32236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "d2a9687c5ef845919d9f39774b4305b6"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "6a40276161f0dc7b29121a2ab1254624"
server_key_iv: "93a8479ac1d34f33534f1abbf87fb25a"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.478300 32236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:01:23.481127 32252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.482914 32236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:23.483062 32236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "d2a9687c5ef845919d9f39774b4305b6"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "6a40276161f0dc7b29121a2ab1254624"
server_key_iv: "93a8479ac1d34f33534f1abbf87fb25a"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.483228 32236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:23.494485 32236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:23.497831 32236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:23.498044 32236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:23.498648 32236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:23.499492 32236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:23.499562 32236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.499626 32236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:23.499675 32236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.508486 32236 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:45701
I20260810 18:01:23.508505 32371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:45701 every 8 connection(s)
I20260810 18:01:23.509373 32236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:23.514422 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 32236
I20260810 18:01:23.514565 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:23.514863 29005 external_mini_cluster.cc:1428] Setting key 406a0d4b4bdaf651033830009b0f6c0e
Aug 10 18:01:23 dist-test-slave-srgv krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:23.524585 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:23.511137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:49881 (local address 127.28.83.126:35259)
0810 18:01:23.511321 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:23.511325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:23.512076 (+ 751us) server_negotiation.cc:407] Connection header received
0810 18:01:23.513595 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:23.513599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:23.513659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:23.513770 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:23.515001 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.515695 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.517396 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.517676 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.517769 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:23.520461 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:23.520468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:23.520474 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:23.520509 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:23.522383 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:23.522906 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:23.522912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:23.522916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:23.523016 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:23.523352 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:23.523358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:23.523361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:23.523612 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:23.523736 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:23.524229 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 18:01:23.524378 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 18:01:23.525307 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:23.511486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:35259 (local address 127.28.83.65:49881)
0810 18:01:23.511934 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:23.511975 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:23.513341 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:23.513936 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:23.513943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:23.514297 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:23.514853 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:23.514868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.515819 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.515822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:23.517265 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:23.517280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.517842 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.517849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:23.518083 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:23.518822 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:23.518837 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:23.520284 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:23.522509 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:23.522515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:23.522528 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:23.522801 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:23.523111 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:23.523113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:23.523114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:23.523250 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:23.523733 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:23.523738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:23.523970 (+ 232us) client_negotiation.cc:769] Sending connection context
0810 18:01:23.524138 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 18:01:23.524406 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:01:23.526625 32372 heartbeater.cc:347] Connected to a master server at 127.28.83.126:35259
I20260810 18:01:23.526901 32372 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:23.527438 32372 heartbeater.cc:511] Master 127.28.83.126:35259 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:23.528941 32172 ts_manager.cc:194] Registered new tserver with Master: d2a9687c5ef845919d9f39774b4305b6 (127.28.83.65:45701)
I20260810 18:01:23.530108 32172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:49881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:23.545054 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35259
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:23.657045 32377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:23.657351 32377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:23.657451 32377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:23.661547 32377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:23.661651 32377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:23.661747 32377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:23.667186 32377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:23.668326 32377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:23.669375 32377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:01:23.679250 32377 server_base.cc:1034] running on GCE node
W20260810 18:01:23.679203 32384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:23.679210 32383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:23.679200 32386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:23.679965 32377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:23.680574 32377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:23.681733 32377 hybrid_clock.cc:648] HybridClock initialized: now 1786384883681693 us; error 53 us; skew 500 ppm
Aug 10 18:01:23 dist-test-slave-srgv krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:23.684442 32377 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:23.685288 32377 webserver.cc:460] Webserver started at http://127.28.83.66:45279/ using document root <none> and password file <none>
I20260810 18:01:23.685878 32377 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:23.685942 32377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:23.686107 32377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:23.688216 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "1926915fe3134cb78165be055278f93c"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "efb74653b5861c4f5fe670cc4d690821"
server_key_iv: "d6d36fb74a1620e5d6fef16c556b4637"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.688848 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "1926915fe3134cb78165be055278f93c"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "efb74653b5861c4f5fe670cc4d690821"
server_key_iv: "d6d36fb74a1620e5d6fef16c556b4637"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.692624 32377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:01:23.695147 32395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.696543 32377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 18:01:23.696727 32377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "1926915fe3134cb78165be055278f93c"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "efb74653b5861c4f5fe670cc4d690821"
server_key_iv: "d6d36fb74a1620e5d6fef16c556b4637"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.696861 32377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:23.716110 32377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:23.719125 32377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:23.719352 32377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:23.719976 32377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:23.720893 32377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:23.720966 32377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.721036 32377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:23.721081 32377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.730964 32377 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:44541
I20260810 18:01:23.731007 32517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:44541 every 8 connection(s)
I20260810 18:01:23.731954 32377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:23.732116 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 32377
I20260810 18:01:23.732244 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:23.732604 29005 external_mini_cluster.cc:1428] Setting key c59d6c799fac366575cc5ae66743220b
Aug 10 18:01:23 dist-test-slave-srgv krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:01:23.748600 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:23.734075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:36851 (local address 127.28.83.126:35259)
0810 18:01:23.734227 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:23.734231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:23.735184 (+ 953us) server_negotiation.cc:407] Connection header received
0810 18:01:23.736566 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:23.736569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:23.736636 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:23.736743 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:23.738025 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.738564 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.739733 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.739972 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.740089 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:23.742830 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:23.742837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:23.742842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:23.742880 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:23.744740 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:23.745379 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:23.745385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:23.745391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:23.745527 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:23.747363 (+ 1836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:23.747366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:23.747368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:23.747581 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:23.747707 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:23.748273 (+ 566us) server_negotiation.cc:299] Negotiation successful
0810 18:01:23.748422 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:01:23.749531 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:23.734482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:35259 (local address 127.28.83.66:36851)
0810 18:01:23.735008 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:23.735052 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:23.736348 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:23.736927 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:23.736934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:23.737304 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:23.737872 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:23.737886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.738705 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.738708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:23.739597 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:23.739607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.740147 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.740151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:23.740321 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:23.740981 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:23.740998 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:23.742614 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:23.744855 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:23.744861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:23.744875 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:23.745216 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:23.747075 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:23.747078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:23.747080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:23.747247 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:23.747714 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:23.747725 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:23.748008 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 18:01:23.748215 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 18:01:23.748574 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":162,"threads_started":1}
I20260810 18:01:23.750939 32518 heartbeater.cc:347] Connected to a master server at 127.28.83.126:35259
I20260810 18:01:23.751201 32518 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:23.751732 32518 heartbeater.cc:511] Master 127.28.83.126:35259 requested a full tablet report, sending...
I20260810 18:01:23.752943 32172 ts_manager.cc:194] Registered new tserver with Master: 1926915fe3134cb78165be055278f93c (127.28.83.66:44541)
I20260810 18:01:23.753832 32172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:36851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:01:23.764066 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35259
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:23.879258 32523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:23.879532 32523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:23.879616 32523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:23.883810 32523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:23.883882 32523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:23.883944 32523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:23.889603 32523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:23.890759 32523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:23.891844 32523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:23.901787 32529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:23.901940 32531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:23.901795 32528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:23.902199 32523 server_base.cc:1034] running on GCE node
I20260810 18:01:23.902601 32523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:23.903256 32523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:23.904419 32523 hybrid_clock.cc:648] HybridClock initialized: now 1786384883904406 us; error 25 us; skew 500 ppm
Aug 10 18:01:23 dist-test-slave-srgv krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:23.911890 32523 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:23.912824 32523 webserver.cc:460] Webserver started at http://127.28.83.67:45839/ using document root <none> and password file <none>
I20260810 18:01:23.913357 32523 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:23.913476 32523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:23.913667 32523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:23.915676 32523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "e8311e7c104f4eb4b82df402b09a3c77"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "c314813326ad804bfdea80dda8d7ec59"
server_key_iv: "097a63fbfbc9a5876c26903e44c05a92"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.916201 32523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "e8311e7c104f4eb4b82df402b09a3c77"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "c314813326ad804bfdea80dda8d7ec59"
server_key_iv: "097a63fbfbc9a5876c26903e44c05a92"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.920056 32523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:01:23.922435 32539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.923663 32523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:23.923810 32523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "e8311e7c104f4eb4b82df402b09a3c77"
format_stamp: "Formatted at 2026-08-10 18:01:23 on dist-test-slave-srgv"
server_key: "c314813326ad804bfdea80dda8d7ec59"
server_key_iv: "097a63fbfbc9a5876c26903e44c05a92"
server_key_version: "encryptionkey@0"
I20260810 18:01:23.923924 32523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:23.954795 32523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:23.959326 32523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:23.959582 32523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:23.960246 32523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:23.961160 32523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:23.961272 32523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.961372 32523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:23.961453 32523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:23.974761 32523 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:38407
I20260810 18:01:23.974803 32658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:38407 every 8 connection(s)
I20260810 18:01:23.976236 32523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:23.981979 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 32523
I20260810 18:01:23.982134 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:23.982496 29005 external_mini_cluster.cc:1428] Setting key e93eab190c87aa61d7c0aaf782fdc673
Aug 10 18:01:23 dist-test-slave-srgv krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:23.993477 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:23.980655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:58333 (local address 127.28.83.126:35259)
0810 18:01:23.980824 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:23.980831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:23.980853 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:01:23.981694 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:23.981697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:23.981759 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:23.981843 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:23.983294 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.984123 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.986004 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.986237 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.986355 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:23.989651 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:23.989658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:23.989665 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:23.989692 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:23.991484 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:23.992026 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:23.992028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:23.992030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:23.992108 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:23.992429 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:23.992431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:23.992432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:23.992610 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:23.992684 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:23.993181 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 18:01:23.993300 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:01:23.994405 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:23.979382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:35259 (local address 127.28.83.67:58333)
0810 18:01:23.979950 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:23.979994 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:23.981458 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:23.982039 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:23.982051 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:23.982560 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:23.983164 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:23.983183 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.984285 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.984289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:23.985889 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:23.985901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:23.986400 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:23.986404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:23.986593 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:23.987346 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:23.987363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:23.989437 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:23.991614 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:23.991619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:23.991635 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:23.991923 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:23.992191 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:23.992193 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:23.992194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:23.992339 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:23.992700 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:23.992705 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:23.992961 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 18:01:23.993123 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 18:01:23.993400 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":169,"threads_started":1}
I20260810 18:01:23.995656 32659 heartbeater.cc:347] Connected to a master server at 127.28.83.126:35259
I20260810 18:01:23.995925 32659 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:23.996587 32659 heartbeater.cc:511] Master 127.28.83.126:35259 requested a full tablet report, sending...
I20260810 18:01:23.997764 32172 ts_manager.cc:194] Registered new tserver with Master: e8311e7c104f4eb4b82df402b09a3c77 (127.28.83.67:38407)
I20260810 18:01:23.998381 32172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:58333
I20260810 18:01:24.008829 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:01:24 dist-test-slave-srgv krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384884, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 18:01:24 dist-test-slave-srgv krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384884, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:24.035238 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:24.026268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.28.83.126:35259)
0810 18:01:24.026444 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:24.026449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:24.026587 (+ 138us) server_negotiation.cc:407] Connection header received
0810 18:01:24.026777 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:24.026780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:24.026825 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:24.026915 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:24.027719 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:24.028244 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:24.029146 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:24.029370 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:24.029550 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:24.032017 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:24.032023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:24.032031 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:24.032057 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:24.033470 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:24.033910 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:24.033914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:24.033917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:24.034003 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:24.034289 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:24.034294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:24.034295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:24.034543 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:24.034694 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:24.034941 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 18:01:24.035070 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 18:01:24.035673 32172 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41872
I20260810 18:01:24.036635 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 32236
I20260810 18:01:24.044855 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 32377
I20260810 18:01:24.052714 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 32523
I20260810 18:01:24.059587 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 32140
2026-08-10T18:01:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2205 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:24 dist-test-slave-srgv krb5kdc[32671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:24 dist-test-slave-srgv krb5kdc[32671](info): set up 2 sockets
Aug 10 18:01:24 dist-test-slave-srgv krb5kdc[32671](info): commencing operation
krb5kdc: starting...
W20260810 18:01:25.124414 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:25 dist-test-slave-srgv krb5kdc[32671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:25.253024 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:46595
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:46595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:25.367169 32684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:25.367492 32684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:25.367575 32684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:25.371649 32684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:25.371753 32684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:25.371793 32684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:25.371865 32684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:25.371904 32684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:25.377254 32684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:46595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:25.378418 32684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:25.379487 32684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:25.388175 32694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:25.388170 32684 server_base.cc:1034] running on GCE node
W20260810 18:01:25.388211 32690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:25.388175 32692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:25.388957 32684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:25.390102 32684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:25.391271 32684 hybrid_clock.cc:648] HybridClock initialized: now 1786384885391259 us; error 26 us; skew 500 ppm
Aug 10 18:01:25 dist-test-slave-srgv krb5kdc[32671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384885, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:25.394098 32684 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:25.395030 32684 webserver.cc:460] Webserver started at http://127.28.83.126:37721/ using document root <none> and password file <none>
I20260810 18:01:25.395540 32684 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:25.395601 32684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:25.395776 32684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:25.397871 32684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "116dd8baf3c04afe88c88dbab4cca81b"
format_stamp: "Formatted at 2026-08-10 18:01:25 on dist-test-slave-srgv"
server_key: "19e70dcd1d60313421e4df92c792bd41"
server_key_iv: "e1a4008a72960fc91b7a71ead2b6466d"
server_key_version: "encryptionkey@0"
I20260810 18:01:25.398398 32684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "116dd8baf3c04afe88c88dbab4cca81b"
format_stamp: "Formatted at 2026-08-10 18:01:25 on dist-test-slave-srgv"
server_key: "19e70dcd1d60313421e4df92c792bd41"
server_key_iv: "e1a4008a72960fc91b7a71ead2b6466d"
server_key_version: "encryptionkey@0"
I20260810 18:01:25.402171 32684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:01:25.404608 32701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:25.405946 32684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:25.406049 32684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "116dd8baf3c04afe88c88dbab4cca81b"
format_stamp: "Formatted at 2026-08-10 18:01:25 on dist-test-slave-srgv"
server_key: "19e70dcd1d60313421e4df92c792bd41"
server_key_iv: "e1a4008a72960fc91b7a71ead2b6466d"
server_key_version: "encryptionkey@0"
I20260810 18:01:25.406178 32684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:25.424559 32684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:25.427465 32684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:25.427670 32684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:25.435045 32684 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:46595
I20260810 18:01:25.435071 32754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:46595 every 8 connection(s)
I20260810 18:01:25.435984 32684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:25.438789 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:25.440264 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 32684
I20260810 18:01:25.440403 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:25.440742 29005 external_mini_cluster.cc:1428] Setting key 33cd27e7374a1b1e0bcef5b8edb8976b
I20260810 18:01:25.445258 32755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b: Bootstrap starting.
I20260810 18:01:25.447572 32755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:25.448468 32755 log.cc:824] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:01:25 dist-test-slave-srgv krb5kdc[32671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:25.450575 32755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b: No bootstrap required, opened a new log
I20260810 18:01:25.454332 32755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116dd8baf3c04afe88c88dbab4cca81b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46595 } }
I20260810 18:01:25.454612 32755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:25.454686 32755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116dd8baf3c04afe88c88dbab4cca81b, State: Initialized, Role: FOLLOWER
I20260810 18:01:25.455178 32755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116dd8baf3c04afe88c88dbab4cca81b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46595 } }
I20260810 18:01:25.455295 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:25.442283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.28.83.126:46595)
0810 18:01:25.442740 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:25.442755 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:25.442792 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:01:25.443405 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:25.443421 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:25.443749 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:25.444097 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:25.444950 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.446007 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.446874 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.447350 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.447521 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:25.450313 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:25.450318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:25.450331 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:25.450373 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:25.452174 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:25.452669 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:25.452673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:25.452677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:25.452790 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:25.453109 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:25.453113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:25.453115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:25.453586 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:25.453806 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:25.454112 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 18:01:25.454407 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":148,"threads_started":1}
I20260810 18:01:25.455331 32755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:25.455475 32755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:25.455582 32755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:25.456637 32755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116dd8baf3c04afe88c88dbab4cca81b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46595 } }
I20260810 18:01:25.456988 32755 leader_election.cc:302] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116dd8baf3c04afe88c88dbab4cca81b; no voters:
I20260810 18:01:25.457320 32755 leader_election.cc:288] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:25.457492 32761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:25.457764 32761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [term 1 LEADER]: Becoming Leader. State: Replica: 116dd8baf3c04afe88c88dbab4cca81b, State: Running, Role: LEADER
I20260810 18:01:25.458114 32761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116dd8baf3c04afe88c88dbab4cca81b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46595 } }
I20260810 18:01:25.458532 32755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:25.459723 32762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "116dd8baf3c04afe88c88dbab4cca81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116dd8baf3c04afe88c88dbab4cca81b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46595 } } }
I20260810 18:01:25.459858 32762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:25.460217 32763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116dd8baf3c04afe88c88dbab4cca81b. Latest consensus state: current_term: 1 leader_uuid: "116dd8baf3c04afe88c88dbab4cca81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116dd8baf3c04afe88c88dbab4cca81b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46595 } } }
I20260810 18:01:25.460378 32763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:25.461247 303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:25.463912 303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:25.469821 303 catalog_manager.cc:1347] Generated new cluster ID: 5b275af019b94069bd345c7e99ccdbb6
I20260810 18:01:25.469895 303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:25.486635 303 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:25.487936 303 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:25.500141 303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 116dd8baf3c04afe88c88dbab4cca81b: Generated new TSK 0
I20260810 18:01:25.500976 303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:25.538882 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46595
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:25.652122 315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:25.652460 315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:25.652545 315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:25.656586 315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:25.656695 315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:25.656791 315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:25.662513 315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:25.663682 315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:25.664789 315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:25.675022 321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:25.675194 315 server_base.cc:1034] running on GCE node
W20260810 18:01:25.675028 320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:25.675109 323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:25.675823 315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:25.676549 315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:25.677749 315 hybrid_clock.cc:648] HybridClock initialized: now 1786384885677733 us; error 29 us; skew 500 ppm
Aug 10 18:01:25 dist-test-slave-srgv krb5kdc[32671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384885, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:25.680976 315 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:25.681993 315 webserver.cc:460] Webserver started at http://127.28.83.65:34689/ using document root <none> and password file <none>
I20260810 18:01:25.682544 315 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:25.682610 315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:25.682847 315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:25.685032 315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "8d348e89f9484e43a542b104589ca47b"
format_stamp: "Formatted at 2026-08-10 18:01:25 on dist-test-slave-srgv"
server_key: "a0d1dcef0d35c00c6464663d477bc8a0"
server_key_iv: "b736308a27a17e966c12e901f6c11e8c"
server_key_version: "encryptionkey@0"
I20260810 18:01:25.685699 315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "8d348e89f9484e43a542b104589ca47b"
format_stamp: "Formatted at 2026-08-10 18:01:25 on dist-test-slave-srgv"
server_key: "a0d1dcef0d35c00c6464663d477bc8a0"
server_key_iv: "b736308a27a17e966c12e901f6c11e8c"
server_key_version: "encryptionkey@0"
I20260810 18:01:25.689841 315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:01:25.692443 330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:25.693817 315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:25.693980 315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "8d348e89f9484e43a542b104589ca47b"
format_stamp: "Formatted at 2026-08-10 18:01:25 on dist-test-slave-srgv"
server_key: "a0d1dcef0d35c00c6464663d477bc8a0"
server_key_iv: "b736308a27a17e966c12e901f6c11e8c"
server_key_version: "encryptionkey@0"
I20260810 18:01:25.694123 315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:25.728097 315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:25.731120 315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:25.731333 315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:25.731937 315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:25.732831 315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:25.732903 315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:25.732977 315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:25.733031 315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:25.742399 315 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:35251
I20260810 18:01:25.742429 449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:35251 every 8 connection(s)
I20260810 18:01:25.743397 315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:25.746678 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 315
I20260810 18:01:25.746829 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:25.747216 29005 external_mini_cluster.cc:1428] Setting key 8afbf6c5271fea264e4e4c176d51e28a
Aug 10 18:01:25 dist-test-slave-srgv krb5kdc[32671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384885, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:01:25.759357 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:25.745315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:47259 (local address 127.28.83.126:46595)
0810 18:01:25.745488 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:25.745492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:25.746353 (+ 861us) server_negotiation.cc:407] Connection header received
0810 18:01:25.748288 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:25.748292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:25.748348 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:25.748435 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:25.749874 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.750510 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.751725 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.752167 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.752297 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:25.755026 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:25.755033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:25.755038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:25.755078 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:25.756955 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:25.757591 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:25.757594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:25.757596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:25.757691 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:25.758097 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:25.758132 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:25.758138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:25.758370 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:25.758452 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:25.758996 (+ 544us) server_negotiation.cc:299] Negotiation successful
0810 18:01:25.759136 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:01:25.760326 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:25.745719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:46595 (local address 127.28.83.65:47259)
0810 18:01:25.746184 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:25.746231 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:25.748041 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:25.748665 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:25.748674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:25.749122 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:25.749716 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:25.749731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.750665 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.750668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:25.751585 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:25.751595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.752332 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.752334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:25.752483 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:25.753168 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:25.753185 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:25.754822 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:25.757101 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:25.757106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:25.757120 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:25.757460 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:25.757784 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:25.757786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:25.757787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:25.757992 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:25.758467 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:25.758473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:25.758737 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 18:01:25.758933 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 18:01:25.759424 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:25.761858 450 heartbeater.cc:347] Connected to a master server at 127.28.83.126:46595
I20260810 18:01:25.762151 450 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:25.762701 450 heartbeater.cc:511] Master 127.28.83.126:46595 requested a full tablet report, sending...
I20260810 18:01:25.764617 32718 ts_manager.cc:194] Registered new tserver with Master: 8d348e89f9484e43a542b104589ca47b (127.28.83.65:35251)
I20260810 18:01:25.766446 32718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:47259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:25.780431 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46595
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:25.896673 455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:25.896955 455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:25.897049 455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:25.901314 455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:25.901381 455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:25.901456 455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:25.906916 455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:25.907951 455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:25.908921 455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:25.918622 460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:25.918799 455 server_base.cc:1034] running on GCE node
W20260810 18:01:25.918622 461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:25.918625 463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:25.919334 455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:25.919973 455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:25.921166 455 hybrid_clock.cc:648] HybridClock initialized: now 1786384885921152 us; error 26 us; skew 500 ppm
Aug 10 18:01:25 dist-test-slave-srgv krb5kdc[32671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384885, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:25.924117 455 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:25.925076 455 webserver.cc:460] Webserver started at http://127.28.83.66:41149/ using document root <none> and password file <none>
I20260810 18:01:25.925740 455 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:25.925833 455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:25.926076 455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:25.928117 455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "29d234e9c1d9446a9a5ece5e14276d4e"
format_stamp: "Formatted at 2026-08-10 18:01:25 on dist-test-slave-srgv"
server_key: "c1ebcceedc3fa323479205bb0b37f6bf"
server_key_iv: "68996cb377848054c294f4fcbd2d416d"
server_key_version: "encryptionkey@0"
I20260810 18:01:25.928718 455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "29d234e9c1d9446a9a5ece5e14276d4e"
format_stamp: "Formatted at 2026-08-10 18:01:25 on dist-test-slave-srgv"
server_key: "c1ebcceedc3fa323479205bb0b37f6bf"
server_key_iv: "68996cb377848054c294f4fcbd2d416d"
server_key_version: "encryptionkey@0"
I20260810 18:01:25.932605 455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:01:25.935190 470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:25.936509 455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:25.936785 455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "29d234e9c1d9446a9a5ece5e14276d4e"
format_stamp: "Formatted at 2026-08-10 18:01:25 on dist-test-slave-srgv"
server_key: "c1ebcceedc3fa323479205bb0b37f6bf"
server_key_iv: "68996cb377848054c294f4fcbd2d416d"
server_key_version: "encryptionkey@0"
I20260810 18:01:25.936928 455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:25.950467 455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:25.954777 455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:25.954990 455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:25.955637 455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:25.956521 455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:25.956569 455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:25.956645 455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:25.956679 455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:25.965973 455 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:35949
I20260810 18:01:25.966053 587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:35949 every 8 connection(s)
I20260810 18:01:25.966938 455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:25.967819 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 455
I20260810 18:01:25.967908 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:25.968209 29005 external_mini_cluster.cc:1428] Setting key ebc1e6c4f61589096db82f91211ddc95
Aug 10 18:01:25 dist-test-slave-srgv krb5kdc[32671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384885, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:01:25.982908 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:25.969607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:36445 (local address 127.28.83.126:46595)
0810 18:01:25.969764 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:25.969769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:25.970581 (+ 812us) server_negotiation.cc:407] Connection header received
0810 18:01:25.972332 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:25.972335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:25.972380 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:25.972442 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:25.973899 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.974441 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.975623 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.975863 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.975972 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:25.978826 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:25.978829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:25.978831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:25.978857 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:25.980356 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:25.981047 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:25.981050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:25.981051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:25.981130 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:25.981567 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:25.981569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:25.981570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:25.981754 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:25.981831 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:25.982553 (+ 722us) server_negotiation.cc:299] Negotiation successful
0810 18:01:25.982692 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:01:25.983760 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:25.969401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:46595 (local address 127.28.83.66:36445)
0810 18:01:25.970440 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:25.970482 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:25.972068 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:25.972674 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:25.972685 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:25.973179 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:25.973744 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:25.973759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.974584 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.974594 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:25.975484 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:25.975493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:25.976031 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:25.976035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:25.976208 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:25.976941 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:25.976959 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:25.978650 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:25.980466 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:25.980471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:25.980485 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:25.980940 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:25.981234 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:25.981238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:25.981241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:25.981468 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:25.981864 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:25.981874 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:25.982270 (+ 396us) client_negotiation.cc:769] Sending connection context
0810 18:01:25.982484 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 18:01:25.982865 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":157,"threads_started":1}
I20260810 18:01:25.985098 588 heartbeater.cc:347] Connected to a master server at 127.28.83.126:46595
I20260810 18:01:25.985363 588 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:25.985913 588 heartbeater.cc:511] Master 127.28.83.126:46595 requested a full tablet report, sending...
I20260810 18:01:25.986996 32718 ts_manager.cc:194] Registered new tserver with Master: 29d234e9c1d9446a9a5ece5e14276d4e (127.28.83.66:35949)
I20260810 18:01:25.987584 32718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:36445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:01:25.996532 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46595
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:26.110687 594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:26.110945 594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:26.110988 594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:26.115029 594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:26.115082 594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:26.115142 594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:26.120191 594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:26.121160 594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:26.122104 594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:26.131349 602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:26.131367 600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:26.131356 599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:26.131762 594 server_base.cc:1034] running on GCE node
I20260810 18:01:26.132149 594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:26.132776 594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:26.133966 594 hybrid_clock.cc:648] HybridClock initialized: now 1786384886133955 us; error 26 us; skew 500 ppm
Aug 10 18:01:26 dist-test-slave-srgv krb5kdc[32671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384886, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:26.136623 594 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:26.137552 594 webserver.cc:460] Webserver started at http://127.28.83.67:36895/ using document root <none> and password file <none>
I20260810 18:01:26.138103 594 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:26.138195 594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:26.138422 594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:26.140509 594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "7cbceb1f5823403fb4eb50f1761b4215"
format_stamp: "Formatted at 2026-08-10 18:01:26 on dist-test-slave-srgv"
server_key: "bf9d5e48af0b672df9d0f477727d917a"
server_key_iv: "c70868b334822963bb59d9d957beed2e"
server_key_version: "encryptionkey@0"
I20260810 18:01:26.141191 594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "7cbceb1f5823403fb4eb50f1761b4215"
format_stamp: "Formatted at 2026-08-10 18:01:26 on dist-test-slave-srgv"
server_key: "bf9d5e48af0b672df9d0f477727d917a"
server_key_iv: "c70868b334822963bb59d9d957beed2e"
server_key_version: "encryptionkey@0"
I20260810 18:01:26.144970 594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 18:01:26.147478 609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:26.148706 594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:26.148849 594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "7cbceb1f5823403fb4eb50f1761b4215"
format_stamp: "Formatted at 2026-08-10 18:01:26 on dist-test-slave-srgv"
server_key: "bf9d5e48af0b672df9d0f477727d917a"
server_key_iv: "c70868b334822963bb59d9d957beed2e"
server_key_version: "encryptionkey@0"
I20260810 18:01:26.148968 594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:26.166090 594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:26.169142 594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:26.169377 594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:26.170071 594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:26.170948 594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:26.171034 594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:26.171133 594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:26.171185 594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:26.180068 594 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:36905
I20260810 18:01:26.180087 726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:36905 every 8 connection(s)
I20260810 18:01:26.181123 594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:26.183594 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 594
I20260810 18:01:26.183743 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:26.184049 29005 external_mini_cluster.cc:1428] Setting key 95b7746285214d07d3fade5d5857bb50
Aug 10 18:01:26 dist-test-slave-srgv krb5kdc[32671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384886, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:26.195745 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:26.183067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:58507 (local address 127.28.83.126:46595)
0810 18:01:26.183252 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:26.183257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:26.184138 (+ 881us) server_negotiation.cc:407] Connection header received
0810 18:01:26.185811 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:26.185815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:26.185879 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:26.185970 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:26.187166 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:26.187689 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:26.188780 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:26.188999 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:26.189086 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:26.191650 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:26.191653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:26.191655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:26.191679 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:26.193557 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:26.194096 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:26.194099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:26.194101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:26.194190 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:26.194544 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:26.194547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:26.194549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:26.194831 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:26.194945 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:26.195436 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 18:01:26.195564 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:01:26.196585 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:26.183467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:46595 (local address 127.28.83.67:58507)
0810 18:01:26.183936 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:26.183991 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:26.185565 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:26.186178 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:26.186188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:26.186560 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:26.187037 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:26.187050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:26.187801 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:26.187803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:26.188670 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:26.188679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:26.189131 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:26.189140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:26.189297 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:26.189955 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:26.189971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:26.191486 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:26.193664 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:26.193670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:26.193688 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:26.193990 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:26.194290 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:26.194293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:26.194296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:26.194444 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:26.194944 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:26.194952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:26.195194 (+ 242us) client_negotiation.cc:769] Sending connection context
0810 18:01:26.195393 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 18:01:26.195723 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":150,"threads_started":1}
I20260810 18:01:26.197734 727 heartbeater.cc:347] Connected to a master server at 127.28.83.126:46595
I20260810 18:01:26.197988 727 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:26.198515 727 heartbeater.cc:511] Master 127.28.83.126:46595 requested a full tablet report, sending...
I20260810 18:01:26.199532 32717 ts_manager.cc:194] Registered new tserver with Master: 7cbceb1f5823403fb4eb50f1761b4215 (127.28.83.67:36905)
I20260810 18:01:26.200122 32717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:58507
I20260810 18:01:26.209991 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:01:26 dist-test-slave-srgv krb5kdc[32671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:01:26 dist-test-slave-srgv krb5kdc[32671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:26.236475 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:26.227136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.28.83.65:35251)
0810 18:01:26.227460 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:26.227475 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:26.227500 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:01:26.227554 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:26.227560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:26.227782 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:26.227909 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:26.228558 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:26.229334 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:26.230174 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:26.230530 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:26.230614 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:26.232964 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:26.232973 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:26.232986 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:26.233010 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:26.234862 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:26.235260 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:26.235265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:26.235267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:26.235349 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:26.235582 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:26.235585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:26.235586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:26.235829 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:26.236008 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:26.236183 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 18:01:26.236316 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":119,"threads_started":1}
Aug 10 18:01:26 dist-test-slave-srgv krb5kdc[32671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:26.248201 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:26.239487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.28.83.126:46595)
0810 18:01:26.239654 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:26.239658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:26.239673 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:01:26.239768 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:26.239770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:26.239809 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:26.239875 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:26.240545 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:26.241057 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:26.241922 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:26.242170 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:26.242255 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:26.244799 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:26.244802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:26.244804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:26.244831 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:26.246411 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:26.246866 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:26.246870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:26.246872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:26.246959 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:26.247244 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:26.247247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:26.247249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:26.247490 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:26.247637 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:26.247889 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 18:01:26.248016 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 18:01:26.248661 32717 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062
I20260810 18:01:26.249294 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 315
I20260810 18:01:26.258509 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 455
I20260810 18:01:26.265260 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 594
I20260810 18:01:26.271660 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 32684
2026-08-10T18:01:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2211 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:26 dist-test-slave-srgv krb5kdc[739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:26 dist-test-slave-srgv krb5kdc[739](info): set up 2 sockets
Aug 10 18:01:26 dist-test-slave-srgv krb5kdc[739](info): commencing operation
krb5kdc: starting...
W20260810 18:01:27.330103 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:27 dist-test-slave-srgv krb5kdc[739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:27.458906 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44217
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:44217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:27.573509 753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:27.573818 753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:27.573864 753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:27.578164 753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:27.578239 753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:27.578259 753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:27.578276 753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:27.578292 753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:27.584218 753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:44217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:27.585525 753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:27.586630 753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:27.595391 759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:27.595391 761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:27.595486 753 server_base.cc:1034] running on GCE node
W20260810 18:01:27.595428 758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:27.596243 753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:27.597292 753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:27.598455 753 hybrid_clock.cc:648] HybridClock initialized: now 1786384887598424 us; error 43 us; skew 500 ppm
Aug 10 18:01:27 dist-test-slave-srgv krb5kdc[739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:27.601248 753 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:27.602046 753 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:27.602113 753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:27.602281 753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:27.604306 753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "3e555e368d854b268e508d0d8e02d231"
format_stamp: "Formatted at 2026-08-10 18:01:27 on dist-test-slave-srgv"
server_key: "18156858327fb33af3540e76472aeb6a"
server_key_iv: "b5b2ff6e152f3b53724ecdb6ea2e8ea9"
server_key_version: "encryptionkey@0"
I20260810 18:01:27.604835 753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "3e555e368d854b268e508d0d8e02d231"
format_stamp: "Formatted at 2026-08-10 18:01:27 on dist-test-slave-srgv"
server_key: "18156858327fb33af3540e76472aeb6a"
server_key_iv: "b5b2ff6e152f3b53724ecdb6ea2e8ea9"
server_key_version: "encryptionkey@0"
I20260810 18:01:27.608533 753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:01:27.610929 767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:27.612121 753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:27.612248 753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "3e555e368d854b268e508d0d8e02d231"
format_stamp: "Formatted at 2026-08-10 18:01:27 on dist-test-slave-srgv"
server_key: "18156858327fb33af3540e76472aeb6a"
server_key_iv: "b5b2ff6e152f3b53724ecdb6ea2e8ea9"
server_key_version: "encryptionkey@0"
I20260810 18:01:27.612354 753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:27.634769 753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:27.637904 753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:27.638126 753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:27.645699 753 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:44217
I20260810 18:01:27.645977 823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:44217 every 8 connection(s)
I20260810 18:01:27.646441 753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:27.649217 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:27.654917 824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231: Bootstrap starting.
I20260810 18:01:27.656507 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 753
I20260810 18:01:27.656643 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:27.656939 29005 external_mini_cluster.cc:1428] Setting key 323f427218559910d97e245c6d00c140
I20260810 18:01:27.657904 824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:27.658856 824 log.cc:824] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:27.660830 824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231: No bootstrap required, opened a new log
I20260810 18:01:27.664563 824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e555e368d854b268e508d0d8e02d231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44217 } }
I20260810 18:01:27.664897 824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:27.664983 824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e555e368d854b268e508d0d8e02d231, State: Initialized, Role: FOLLOWER
I20260810 18:01:27.665517 824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e555e368d854b268e508d0d8e02d231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44217 } }
I20260810 18:01:27.665673 824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:27.665752 824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:27.665853 824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 18:01:27 dist-test-slave-srgv krb5kdc[739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:27.666929 824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e555e368d854b268e508d0d8e02d231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44217 } }
I20260810 18:01:27.667307 824 leader_election.cc:302] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e555e368d854b268e508d0d8e02d231; no voters:
I20260810 18:01:27.667722 824 leader_election.cc:288] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:27.667796 829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:27.668362 829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [term 1 LEADER]: Becoming Leader. State: Replica: 3e555e368d854b268e508d0d8e02d231, State: Running, Role: LEADER
I20260810 18:01:27.668730 829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e555e368d854b268e508d0d8e02d231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44217 } }
I20260810 18:01:27.669178 824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:27.670432 830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e555e368d854b268e508d0d8e02d231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e555e368d854b268e508d0d8e02d231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44217 } } }
I20260810 18:01:27.670584 830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:27.671083 831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e555e368d854b268e508d0d8e02d231. Latest consensus state: current_term: 1 leader_uuid: "3e555e368d854b268e508d0d8e02d231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e555e368d854b268e508d0d8e02d231" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44217 } } }
I20260810 18:01:27.671247 831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:27.673203 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:27.658184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.28.83.126:44217)
0810 18:01:27.658673 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:27.658687 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:27.658722 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:01:27.659603 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:27.659640 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:27.659971 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:27.660303 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:27.661172 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:27.662223 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:27.663136 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:27.663627 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:27.663799 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:27.666849 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:27.666855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:27.666868 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:27.666898 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:27.669020 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:27.669611 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:27.669623 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:27.669628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:27.669746 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:27.670097 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:27.670103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:27.670106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:27.670721 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:27.671820 (+ 1099us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:27.671982 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 18:01:27.672260 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":146,"threads_started":1}
W20260810 18:01:27.674333 844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:01:27.674417 844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:01:27.674487 839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:27.675449 839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:27.681216 839 catalog_manager.cc:1347] Generated new cluster ID: 0eae9e9f1b5c474fb92898d4e15e35d2
I20260810 18:01:27.681311 839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:27.691748 839 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:27.692745 839 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:27.703094 839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231: Generated new TSK 0
I20260810 18:01:27.703806 839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:27.742599 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44217
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:27.859360 850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:27.859612 850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:27.859658 850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:27.863788 850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:27.863863 850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:27.863924 850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:27.869663 850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:27.870875 850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:27.871981 850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:27.881484 855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:27.881497 858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:27.881651 850 server_base.cc:1034] running on GCE node
W20260810 18:01:27.881855 856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:27.882330 850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:27.882982 850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:27.884150 850 hybrid_clock.cc:648] HybridClock initialized: now 1786384887884115 us; error 45 us; skew 500 ppm
Aug 10 18:01:27 dist-test-slave-srgv krb5kdc[739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:27.887116 850 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:27.887873 850 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:27.887938 850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:27.888187 850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:27.890569 850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "026633ee24b24dabaefd36e27de9f44c"
format_stamp: "Formatted at 2026-08-10 18:01:27 on dist-test-slave-srgv"
server_key: "df8d11eb663a38a91f93d3cc03e44b54"
server_key_iv: "4cdaa8b4ad53090abb32dcea134d240d"
server_key_version: "encryptionkey@0"
I20260810 18:01:27.891182 850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "026633ee24b24dabaefd36e27de9f44c"
format_stamp: "Formatted at 2026-08-10 18:01:27 on dist-test-slave-srgv"
server_key: "df8d11eb663a38a91f93d3cc03e44b54"
server_key_iv: "4cdaa8b4ad53090abb32dcea134d240d"
server_key_version: "encryptionkey@0"
I20260810 18:01:27.895000 850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:01:27.897362 864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:27.898597 850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:27.898730 850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "026633ee24b24dabaefd36e27de9f44c"
format_stamp: "Formatted at 2026-08-10 18:01:27 on dist-test-slave-srgv"
server_key: "df8d11eb663a38a91f93d3cc03e44b54"
server_key_iv: "4cdaa8b4ad53090abb32dcea134d240d"
server_key_version: "encryptionkey@0"
I20260810 18:01:27.898838 850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:27.913935 850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:27.917526 850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:27.917739 850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:27.918322 850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:27.919154 850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:27.919201 850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:27.919266 850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:27.919297 850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:27.928279 850 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:43395
I20260810 18:01:27.928335 983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:43395 every 8 connection(s)
I20260810 18:01:27.928936 850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:27.930351 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 850
I20260810 18:01:27.930488 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:27.930853 29005 external_mini_cluster.cc:1428] Setting key f5a73bc14c10128335b9f9e629ce617e
Aug 10 18:01:27 dist-test-slave-srgv krb5kdc[739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:27.945627 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:27.930835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:41015 (local address 127.28.83.126:44217)
0810 18:01:27.930970 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:27.930982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:27.931960 (+ 978us) server_negotiation.cc:407] Connection header received
0810 18:01:27.933951 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:27.933954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:27.934000 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:27.934063 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:27.935403 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:27.936309 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:27.937665 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:27.938119 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:27.938273 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:27.940855 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:27.940859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:27.940862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:27.940896 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:27.942623 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:27.943146 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:27.943149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:27.943150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:27.943258 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:27.943792 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:27.943795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:27.943797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:27.944099 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:27.944206 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:27.945022 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 18:01:27.945273 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:01:27.946482 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:27.931327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44217 (local address 127.28.83.65:41015)
0810 18:01:27.931800 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:27.931854 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:27.933706 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:27.934283 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:27.934297 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:27.934744 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:27.935235 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:27.935249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:27.936446 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:27.936449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:27.937525 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:27.937536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:27.938302 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:27.938305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:27.938449 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:27.939125 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:27.939142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:27.940659 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:27.942737 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:27.942742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:27.942757 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:27.943033 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:27.943385 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:27.943390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:27.943393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:27.943670 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:27.944286 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:27.944294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:27.944666 (+ 372us) client_negotiation.cc:769] Sending connection context
0810 18:01:27.944910 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 18:01:27.945319 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":138,"threads_started":1}
I20260810 18:01:27.947937 984 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44217
I20260810 18:01:27.948233 984 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:27.948776 984 heartbeater.cc:511] Master 127.28.83.126:44217 requested a full tablet report, sending...
I20260810 18:01:27.950562 786 ts_manager.cc:194] Registered new tserver with Master: 026633ee24b24dabaefd36e27de9f44c (127.28.83.65:43395)
I20260810 18:01:27.951962 786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:41015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:27.962754 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44217
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:28.076264 989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:28.076599 989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:28.076685 989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:28.080816 989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:28.080947 989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:28.081049 989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:28.086601 989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:28.087765 989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:28.088811 989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:28.098479 995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:28.098479 997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:28.098453 994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:28.098711 989 server_base.cc:1034] running on GCE node
I20260810 18:01:28.099267 989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:28.099918 989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:28.101090 989 hybrid_clock.cc:648] HybridClock initialized: now 1786384888101073 us; error 29 us; skew 500 ppm
Aug 10 18:01:28 dist-test-slave-srgv krb5kdc[739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384888, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:28.103960 989 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:28.104718 989 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:28.104780 989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:28.104943 989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:28.106974 989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "cd65559549894e18af96e43edffad6d5"
format_stamp: "Formatted at 2026-08-10 18:01:28 on dist-test-slave-srgv"
server_key: "35314b8f3b1182f434be9c750c8ad703"
server_key_iv: "90321b32bb0bdb26de76c7d7136ef2ab"
server_key_version: "encryptionkey@0"
I20260810 18:01:28.107492 989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "cd65559549894e18af96e43edffad6d5"
format_stamp: "Formatted at 2026-08-10 18:01:28 on dist-test-slave-srgv"
server_key: "35314b8f3b1182f434be9c750c8ad703"
server_key_iv: "90321b32bb0bdb26de76c7d7136ef2ab"
server_key_version: "encryptionkey@0"
I20260810 18:01:28.111162 989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 18:01:28.113457 1003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:28.114674 989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:28.114809 989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "cd65559549894e18af96e43edffad6d5"
format_stamp: "Formatted at 2026-08-10 18:01:28 on dist-test-slave-srgv"
server_key: "35314b8f3b1182f434be9c750c8ad703"
server_key_iv: "90321b32bb0bdb26de76c7d7136ef2ab"
server_key_version: "encryptionkey@0"
I20260810 18:01:28.114923 989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:28.125371 989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:28.128182 989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:28.128381 989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:28.128945 989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:28.129830 989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:28.129901 989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:28.129973 989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:28.130025 989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:28.139329 989 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:36469
I20260810 18:01:28.139348 1120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:36469 every 8 connection(s)
I20260810 18:01:28.139988 989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
Aug 10 18:01:28 dist-test-slave-srgv krb5kdc[739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384888, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:28.149931 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 989
I20260810 18:01:28.150067 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:28.150382 29005 external_mini_cluster.cc:1428] Setting key 1f1b61a5113ba8de1e94b65f26a0fd29
I20260810 18:01:28.155049 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:28.141830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:35983 (local address 127.28.83.126:44217)
0810 18:01:28.142001 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:28.142006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:28.142887 (+ 881us) server_negotiation.cc:407] Connection header received
0810 18:01:28.144278 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:28.144282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:28.144335 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:28.144419 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:28.145749 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:28.146275 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:28.147425 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:28.147642 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:28.147718 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:28.150394 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:28.150397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:28.150399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:28.150425 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:28.152363 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:28.153030 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:28.153032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:28.153033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:28.153108 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:28.153555 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:28.153561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:28.153566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:28.153840 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:28.153952 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:28.154611 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 18:01:28.154785 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:01:28.155746 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:28.142262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44217 (local address 127.28.83.66:35983)
0810 18:01:28.142724 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:28.142770 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:28.144016 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:28.144596 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:28.144602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:28.144990 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:28.145593 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:28.145610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:28.146405 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:28.146408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:28.147312 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:28.147321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:28.147781 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:28.147783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:28.147922 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:28.148513 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:28.148530 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:28.150211 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:28.152489 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:28.152495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:28.152513 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:28.152922 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:28.153205 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:28.153211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:28.153216 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:28.153445 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:28.153967 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:28.153978 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:28.154337 (+ 359us) client_negotiation.cc:769] Sending connection context
0810 18:01:28.154501 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 18:01:28.154791 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
I20260810 18:01:28.157183 1121 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44217
I20260810 18:01:28.157485 1121 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:28.158155 1121 heartbeater.cc:511] Master 127.28.83.126:44217 requested a full tablet report, sending...
I20260810 18:01:28.159541 786 ts_manager.cc:194] Registered new tserver with Master: cd65559549894e18af96e43edffad6d5 (127.28.83.66:36469)
I20260810 18:01:28.160889 786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:35983
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
W20260810 18:01:28.800266 29005 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.83.67: real 0.648s user 0.000s sys 0.004s
I20260810 18:01:28.800793 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44217
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:28.912248 1127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:28.912492 1127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:28.912533 1127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:28.916620 1127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:28.916713 1127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:28.916819 1127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:28.922377 1127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:28.923401 1127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:28.924446 1127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:28.933768 1135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:28.933768 1132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:28.933770 1133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:28.934053 1127 server_base.cc:1034] running on GCE node
I20260810 18:01:28.934469 1127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:28.935141 1127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:28.936329 1127 hybrid_clock.cc:648] HybridClock initialized: now 1786384888936315 us; error 28 us; skew 500 ppm
Aug 10 18:01:28 dist-test-slave-srgv krb5kdc[739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384888, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:28.943181 1127 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:28.943996 1127 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:28.944110 1127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:28.944358 1127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:28.946544 1127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "86a9e60d4159425b9dcdcd73cdb50dea"
format_stamp: "Formatted at 2026-08-10 18:01:28 on dist-test-slave-srgv"
server_key: "94366f8f4db7ace907e704fe821d0af0"
server_key_iv: "badbb449d5278320da4b791d3eb73f28"
server_key_version: "encryptionkey@0"
I20260810 18:01:28.947207 1127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "86a9e60d4159425b9dcdcd73cdb50dea"
format_stamp: "Formatted at 2026-08-10 18:01:28 on dist-test-slave-srgv"
server_key: "94366f8f4db7ace907e704fe821d0af0"
server_key_iv: "badbb449d5278320da4b791d3eb73f28"
server_key_version: "encryptionkey@0"
I20260810 18:01:28.951571 1127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:01:28.954062 1141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:28.955014 984 heartbeater.cc:503] Master 127.28.83.126:44217 was elected leader, sending a full tablet report...
I20260810 18:01:28.955448 1127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:28.955593 1127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "86a9e60d4159425b9dcdcd73cdb50dea"
format_stamp: "Formatted at 2026-08-10 18:01:28 on dist-test-slave-srgv"
server_key: "94366f8f4db7ace907e704fe821d0af0"
server_key_iv: "badbb449d5278320da4b791d3eb73f28"
server_key_version: "encryptionkey@0"
I20260810 18:01:28.955729 1127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:28.970068 1127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:28.973069 1127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:28.973227 1127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:28.973824 1127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:28.974596 1127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:28.974642 1127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:28.974681 1127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:28.974712 1127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:28.983716 1127 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:35559
I20260810 18:01:28.983726 1259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:35559 every 8 connection(s)
I20260810 18:01:28.984364 1127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:28.987744 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 1127
I20260810 18:01:28.987860 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:28.988168 29005 external_mini_cluster.cc:1428] Setting key be1c45a5679d86c32dcd2ed4a83720da
Aug 10 18:01:28 dist-test-slave-srgv krb5kdc[739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384888, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:28.998533 1262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:28.986218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:33527 (local address 127.28.83.126:44217)
0810 18:01:28.986474 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:28.986477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:28.987243 (+ 766us) server_negotiation.cc:407] Connection header received
0810 18:01:28.988759 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:28.988763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:28.988827 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:28.988950 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:28.990426 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:28.990975 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:28.992129 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:28.992361 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:28.992480 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:28.995001 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:28.995004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:28.995007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:28.995030 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:28.996369 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:28.996904 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:28.996909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:28.996911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:28.996999 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:28.997345 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:28.997350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:28.997352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:28.997598 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:28.997722 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:28.998218 (+ 496us) server_negotiation.cc:299] Negotiation successful
0810 18:01:28.998347 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260810 18:01:28.999293 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:28.986613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44217 (local address 127.28.83.67:33527)
0810 18:01:28.987060 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:28.987127 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:28.988539 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:28.989155 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:28.989166 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:28.989685 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:28.990271 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:28.990290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:28.991113 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:28.991117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:28.991982 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:28.991992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:28.992531 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:28.992535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:28.992706 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:28.993296 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:28.993312 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:28.994831 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:28.996507 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:28.996512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:28.996527 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:28.996798 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:28.997105 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:28.997108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:28.997109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:28.997246 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:28.997718 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:28.997724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:28.997978 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 18:01:28.998176 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 18:01:28.998490 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260810 18:01:29.000385 1260 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44217
I20260810 18:01:29.000633 1260 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:29.001204 1260 heartbeater.cc:511] Master 127.28.83.126:44217 requested a full tablet report, sending...
I20260810 18:01:29.002300 786 ts_manager.cc:194] Registered new tserver with Master: 86a9e60d4159425b9dcdcd73cdb50dea (127.28.83.67:35559)
I20260810 18:01:29.002904 786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:33527
I20260810 18:01:29.013767 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:01:29.023087 1262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.015739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.28.83.126:44217)
0810 18:01:29.015858 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.015862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.016068 (+ 206us) server_negotiation.cc:407] Connection header received
0810 18:01:29.016168 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.016170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.016245 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.016329 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:29.017037 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.017609 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.018486 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.018712 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.018791 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.020170 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:29.020173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:29.020175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:29.020195 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:29.021757 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:29.022117 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:29.022119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:29.022121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:29.022192 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:29.022421 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:29.022423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:29.022424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:29.022622 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:29.022736 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.022885 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.022975 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:01:29.025871 786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:29.028786 786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:29.043310 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.037955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:43395 (local address 127.0.0.1:54134)
0810 18:01:29.038740 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:29.038754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:29.038851 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:29.039545 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:29.039547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:29.039560 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:29.039626 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.039630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.040676 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.040679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.041889 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.041899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.042730 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.042733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.042838 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.042873 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 18:01:29.042981 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 18:01:29.043115 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":55,"spinlock_wait_cycles":1408,"thread_start_us":89,"threads_started":1}
I20260810 18:01:29.043510 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.038376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.28.83.65:43395)
0810 18:01:29.038998 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.039007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.039029 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:01:29.039105 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.039110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.039284 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.039437 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:29.039749 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.040528 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.042021 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.042592 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.042717 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.042871 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.043119 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.043236 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":73,"threads_started":1}
I20260810 18:01:29.043558 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.037137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:36469 (local address 127.0.0.1:50094)
0810 18:01:29.037898 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:29.037952 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:29.038085 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:29.038772 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:29.038778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:29.038797 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:29.038939 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.038946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.040408 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.040411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.041844 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.041855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.043027 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.043031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.043227 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.043249 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:01:29.043323 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 18:01:29.043390 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":63,"spinlock_wait_cycles":2944,"thread_start_us":145,"threads_started":1}
I20260810 18:01:29.043665 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.038034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.28.83.67:35559)
0810 18:01:29.038568 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.038584 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.038609 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:01:29.038790 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.038799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.038984 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.039157 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:29.039374 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.040457 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.041836 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.042664 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.042790 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.042945 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.043382 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.043491 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":191,"threads_started":1}
I20260810 18:01:29.044080 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.037336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.28.83.66:36469)
0810 18:01:29.037715 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.037722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.038236 (+ 514us) server_negotiation.cc:407] Connection header received
0810 18:01:29.038324 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.038329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.038535 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.038702 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:29.039072 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.040286 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.041980 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.042724 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.042818 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.042971 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.043796 (+ 825us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.043913 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":87,"threads_started":1}
I20260810 18:01:29.044070 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.037558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:35559 (local address 127.0.0.1:47352)
0810 18:01:29.038531 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:29.038544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:29.038635 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:29.039146 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:29.039149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:29.039167 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:29.039250 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.039268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.040623 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.040628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.041714 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.041723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.042951 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.042957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.043092 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.043165 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 18:01:29.043235 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:01:29.043294 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":46,"thread_start_us":91,"threads_started":1}
I20260810 18:01:29.046751 915 tablet_service.cc:1467] Processing CreateTablet for tablet 4f13ea5eb6ca40598e98b38dc212c943 (DEFAULT_TABLE table=test-table [id=fcef7057952b4b0d9ffcb38d189951a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:29.046876 1191 tablet_service.cc:1467] Processing CreateTablet for tablet 4f13ea5eb6ca40598e98b38dc212c943 (DEFAULT_TABLE table=test-table [id=fcef7057952b4b0d9ffcb38d189951a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:29.047186 1053 tablet_service.cc:1467] Processing CreateTablet for tablet 4f13ea5eb6ca40598e98b38dc212c943 (DEFAULT_TABLE table=test-table [id=fcef7057952b4b0d9ffcb38d189951a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:29.047827 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f13ea5eb6ca40598e98b38dc212c943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:29.047960 1191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f13ea5eb6ca40598e98b38dc212c943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:29.048022 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f13ea5eb6ca40598e98b38dc212c943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:29.054956 1281 tablet_bootstrap.cc:492] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: Bootstrap starting.
I20260810 18:01:29.055179 1280 tablet_bootstrap.cc:492] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c: Bootstrap starting.
I20260810 18:01:29.055873 1282 tablet_bootstrap.cc:492] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: Bootstrap starting.
I20260810 18:01:29.057564 1281 tablet_bootstrap.cc:654] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:29.057564 1280 tablet_bootstrap.cc:654] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:29.057650 1282 tablet_bootstrap.cc:654] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:29.058468 1280 log.cc:824] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:29.058495 1281 log.cc:824] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:29.058780 1282 log.cc:824] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:29.060839 1282 tablet_bootstrap.cc:492] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: No bootstrap required, opened a new log
I20260810 18:01:29.060840 1281 tablet_bootstrap.cc:492] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: No bootstrap required, opened a new log
I20260810 18:01:29.060840 1280 tablet_bootstrap.cc:492] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c: No bootstrap required, opened a new log
I20260810 18:01:29.061072 1281 ts_tablet_manager.cc:1397] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 18:01:29.061072 1280 ts_tablet_manager.cc:1397] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260810 18:01:29.061076 1282 ts_tablet_manager.cc:1397] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:01:29.064014 1282 raft_consensus.cc:348] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } }
I20260810 18:01:29.064337 1282 raft_consensus.cc:374] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:29.064399 1282 raft_consensus.cc:729] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a9e60d4159425b9dcdcd73cdb50dea, State: Initialized, Role: FOLLOWER
I20260810 18:01:29.064926 1282 consensus_queue.cc:260] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } }
I20260810 18:01:29.065675 1260 heartbeater.cc:503] Master 127.28.83.126:44217 was elected leader, sending a full tablet report...
I20260810 18:01:29.065946 1280 raft_consensus.cc:348] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } }
I20260810 18:01:29.066223 1280 raft_consensus.cc:374] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:29.066284 1280 raft_consensus.cc:729] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026633ee24b24dabaefd36e27de9f44c, State: Initialized, Role: FOLLOWER
I20260810 18:01:29.066206 1281 raft_consensus.cc:348] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } }
I20260810 18:01:29.066444 1281 raft_consensus.cc:374] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:29.066501 1281 raft_consensus.cc:729] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd65559549894e18af96e43edffad6d5, State: Initialized, Role: FOLLOWER
I20260810 18:01:29.066726 1280 consensus_queue.cc:260] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } }
I20260810 18:01:29.066856 1281 consensus_queue.cc:260] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } }
I20260810 18:01:29.067198 1282 ts_tablet_manager.cc:1428] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:01:29.067719 1281 ts_tablet_manager.cc:1428] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260810 18:01:29.067741 1121 heartbeater.cc:503] Master 127.28.83.126:44217 was elected leader, sending a full tablet report...
I20260810 18:01:29.069252 1280 ts_tablet_manager.cc:1428] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 18:01:29.129485 1288 raft_consensus.cc:482] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:29.129737 1288 raft_consensus.cc:504] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } }
I20260810 18:01:29.130954 1288 leader_election.cc:288] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 026633ee24b24dabaefd36e27de9f44c (127.28.83.65:43395), 86a9e60d4159425b9dcdcd73cdb50dea (127.28.83.67:35559)
I20260810 18:01:29.135121 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.131400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:40627 (local address 127.28.83.67:35559)
0810 18:01:29.131533 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.131538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.131801 (+ 263us) server_negotiation.cc:407] Connection header received
0810 18:01:29.131927 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.131930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.131992 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.132072 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:29.132406 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.132882 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.134112 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.134516 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.134644 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.134698 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.134893 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.134972 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:01:29.135136 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.131385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:58799 (local address 127.28.83.65:43395)
0810 18:01:29.131527 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.131534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.131739 (+ 205us) server_negotiation.cc:407] Connection header received
0810 18:01:29.131867 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.131873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.131928 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.131985 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:29.132370 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.132881 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.134128 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.134540 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.134677 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.134730 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.134949 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.135031 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:01:29.135149 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.131241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:43395 (local address 127.28.83.66:58799)
0810 18:01:29.131636 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:29.131648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:29.131746 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:29.132029 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:29.132034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:29.132055 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:29.132247 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.132254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.133000 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.133004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.133972 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.133982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.134687 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.134691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.134797 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.134814 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 18:01:29.134913 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 18:01:29.134977 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":68,"spinlock_wait_cycles":28416,"thread_start_us":142,"threads_started":1}
I20260810 18:01:29.135156 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.131261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:35559 (local address 127.28.83.66:40627)
0810 18:01:29.131658 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:29.131669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:29.131809 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:29.132100 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:29.132102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:29.132145 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:29.132267 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.132272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.133007 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.133011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.133968 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:29.133978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.134643 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.134646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:29.134753 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.134783 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:01:29.134846 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 18:01:29.134939 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":75,"spinlock_wait_cycles":768,"thread_start_us":116,"threads_started":1}
I20260810 18:01:29.135790 936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f13ea5eb6ca40598e98b38dc212c943" candidate_uuid: "cd65559549894e18af96e43edffad6d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026633ee24b24dabaefd36e27de9f44c" is_pre_election: true
I20260810 18:01:29.136122 936 raft_consensus.cc:2454] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd65559549894e18af96e43edffad6d5 in term 0.
I20260810 18:01:29.136089 1211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f13ea5eb6ca40598e98b38dc212c943" candidate_uuid: "cd65559549894e18af96e43edffad6d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" is_pre_election: true
I20260810 18:01:29.136360 1211 raft_consensus.cc:2454] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd65559549894e18af96e43edffad6d5 in term 0.
I20260810 18:01:29.136633 1007 leader_election.cc:302] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026633ee24b24dabaefd36e27de9f44c, cd65559549894e18af96e43edffad6d5; no voters:
I20260810 18:01:29.136938 1288 raft_consensus.cc:2781] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:29.137040 1288 raft_consensus.cc:482] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:29.137107 1288 raft_consensus.cc:3037] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:29.138216 1288 raft_consensus.cc:504] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } }
I20260810 18:01:29.138625 1288 leader_election.cc:288] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [CANDIDATE]: Term 1 election: Requested vote from peers 026633ee24b24dabaefd36e27de9f44c (127.28.83.65:43395), 86a9e60d4159425b9dcdcd73cdb50dea (127.28.83.67:35559)
I20260810 18:01:29.139035 936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f13ea5eb6ca40598e98b38dc212c943" candidate_uuid: "cd65559549894e18af96e43edffad6d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026633ee24b24dabaefd36e27de9f44c"
I20260810 18:01:29.139007 1211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f13ea5eb6ca40598e98b38dc212c943" candidate_uuid: "cd65559549894e18af96e43edffad6d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a9e60d4159425b9dcdcd73cdb50dea"
I20260810 18:01:29.139174 1211 raft_consensus.cc:3037] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:29.139168 936 raft_consensus.cc:3037] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:29.140373 1211 raft_consensus.cc:2454] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd65559549894e18af96e43edffad6d5 in term 1.
I20260810 18:01:29.140537 936 raft_consensus.cc:2454] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd65559549894e18af96e43edffad6d5 in term 1.
I20260810 18:01:29.140736 1005 leader_election.cc:302] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86a9e60d4159425b9dcdcd73cdb50dea, cd65559549894e18af96e43edffad6d5; no voters:
I20260810 18:01:29.141004 1288 raft_consensus.cc:2781] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:29.141359 1288 raft_consensus.cc:686] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 1 LEADER]: Becoming Leader. State: Replica: cd65559549894e18af96e43edffad6d5, State: Running, Role: LEADER
W20260810 18:01:29.141615 1122 tablet.cc:2369] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:29.141752 1288 consensus_queue.cc:237] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } }
I20260810 18:01:29.145255 783 catalog_manager.cc:5495] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd65559549894e18af96e43edffad6d5 (127.28.83.66). New cstate: current_term: 1 leader_uuid: "cd65559549894e18af96e43edffad6d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd65559549894e18af96e43edffad6d5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 } health_report { overall_health: UNKNOWN } } }
I20260810 18:01:29.173944 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.170353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.28.83.66:36469)
0810 18:01:29.170519 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.170523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.170538 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:01:29.170715 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.170717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.170775 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.170867 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:29.171098 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.171663 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.172468 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:29.172673 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:29.172756 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:29.173033 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:29.173503 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:29.173592 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.173676 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.173739 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:01:29.180016 1211 raft_consensus.cc:1261] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [term 1 FOLLOWER]: Refusing update from remote peer cd65559549894e18af96e43edffad6d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:29.180016 936 raft_consensus.cc:1261] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c [term 1 FOLLOWER]: Refusing update from remote peer cd65559549894e18af96e43edffad6d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 18:01:29.180737 985 tablet.cc:2369] T 4f13ea5eb6ca40598e98b38dc212c943 P 026633ee24b24dabaefd36e27de9f44c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:29.180809 1291 consensus_queue.cc:1035] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a9e60d4159425b9dcdcd73cdb50dea" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:29.181051 1288 consensus_queue.cc:1035] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026633ee24b24dabaefd36e27de9f44c" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:29.188514 1294 mvcc.cc:205] Tried to move back new op lower bound from 7317032506073276416 to 7317032505927475200. Current Snapshot: MvccSnapshot[applied={T|T < 7317032506073276416}]
I20260810 18:01:29.190176 1295 mvcc.cc:205] Tried to move back new op lower bound from 7317032506073276416 to 7317032505927475200. Current Snapshot: MvccSnapshot[applied={T|T < 7317032506073276416 or (T in {7317032506073276416})}]
I20260810 18:01:29.197732 783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:29.197914 783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:29.200304 783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e555e368d854b268e508d0d8e02d231: Sending DeleteTablet for 3 replicas of tablet 4f13ea5eb6ca40598e98b38dc212c943
I20260810 18:01:29.201112 1053 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f13ea5eb6ca40598e98b38dc212c943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:29 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:50094
I20260810 18:01:29.201236 1191 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f13ea5eb6ca40598e98b38dc212c943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:29 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:47352
I20260810 18:01:29.201668 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 850
I20260810 18:01:29.202888 1305 tablet_replica.cc:330] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: stopping tablet replica
I20260810 18:01:29.203287 1305 raft_consensus.cc:2229] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:29.203712 1305 raft_consensus.cc:2258] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:29.205335 1304 tablet_replica.cc:330] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: stopping tablet replica
I20260810 18:01:29.205495 1305 ts_tablet_manager.cc:1905] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:29.205669 1304 raft_consensus.cc:2229] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:29.206061 1304 raft_consensus.cc:2258] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:29.207587 1304 ts_tablet_manager.cc:1905] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:29.208652 1305 ts_tablet_manager.cc:1918] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:29.208801 1305 log.cc:1196] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/4f13ea5eb6ca40598e98b38dc212c943
I20260810 18:01:29.209223 1305 ts_tablet_manager.cc:1939] T 4f13ea5eb6ca40598e98b38dc212c943 P 86a9e60d4159425b9dcdcd73cdb50dea: Deleting consensus metadata
W20260810 18:01:29.210199 773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:43395 (error 108)
I20260810 18:01:29.210719 1304 ts_tablet_manager.cc:1918] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:29.210819 1304 log.cc:1196] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/4f13ea5eb6ca40598e98b38dc212c943
I20260810 18:01:29.210999 769 catalog_manager.cc:4841] TS 86a9e60d4159425b9dcdcd73cdb50dea (127.28.83.67:35559): tablet 4f13ea5eb6ca40598e98b38dc212c943 (table test-table [id=fcef7057952b4b0d9ffcb38d189951a8]) successfully deleted
I20260810 18:01:29.211146 1304 ts_tablet_manager.cc:1939] T 4f13ea5eb6ca40598e98b38dc212c943 P cd65559549894e18af96e43edffad6d5: Deleting consensus metadata
I20260810 18:01:29.211633 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 989
I20260810 18:01:29.211707 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.211404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:43395 (local address 127.0.0.1:54146)
0810 18:01:29.211551 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:29.211623 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 18:01:29.213312 773 catalog_manager.cc:4584] TS 026633ee24b24dabaefd36e27de9f44c (127.28.83.65:43395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f13ea5eb6ca40598e98b38dc212c943: Network error: Client connection negotiation failed: client connection to 127.28.83.65:43395: connect: Connection refused (error 111)
I20260810 18:01:29.219038 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 1127
I20260810 18:01:29.220983 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.219256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:36469 (local address 127.0.0.1:50110)
0810 18:01:29.220801 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:29.220873 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.66:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1423}
I20260810 18:01:29.225335 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 753
2026-08-10T18:01:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2953 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T18:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:29Z Disabled control of system clock
I20260810 18:01:29.257848 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:41449
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:41449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:01:29.371335 1310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:29.371632 1310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:29.371671 1310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:29.375900 1310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:29.375972 1310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:29.375988 1310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:29.376004 1310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:29.376019 1310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:29.380934 1310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:41449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:29.382166 1310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:29.383675 1310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:29.392643 1316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:29.392637 1315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:29.392643 1318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:29.393036 1310 server_base.cc:1034] running on GCE node
I20260810 18:01:29.393745 1310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:29.394855 1310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:29.396040 1310 hybrid_clock.cc:648] HybridClock initialized: now 1786384889396001 us; error 51 us; skew 500 ppm
I20260810 18:01:29.398126 1310 webserver.cc:460] Webserver started at http://127.28.83.126:37619/ using document root <none> and password file <none>
I20260810 18:01:29.398690 1310 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:29.398751 1310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:29.398922 1310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:29.400974 1310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "bb89122517b44c63ad38f68362c13c6f"
format_stamp: "Formatted at 2026-08-10 18:01:29 on dist-test-slave-srgv"
server_key: "39ce67cdd247842623cb245b879ba547"
server_key_iv: "4c8c89bdff8da548c1c69c1669aa677b"
server_key_version: "encryptionkey@0"
I20260810 18:01:29.401592 1310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "bb89122517b44c63ad38f68362c13c6f"
format_stamp: "Formatted at 2026-08-10 18:01:29 on dist-test-slave-srgv"
server_key: "39ce67cdd247842623cb245b879ba547"
server_key_iv: "4c8c89bdff8da548c1c69c1669aa677b"
server_key_version: "encryptionkey@0"
I20260810 18:01:29.405279 1310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:29.407836 1325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:29.409090 1310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:29.409235 1310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "bb89122517b44c63ad38f68362c13c6f"
format_stamp: "Formatted at 2026-08-10 18:01:29 on dist-test-slave-srgv"
server_key: "39ce67cdd247842623cb245b879ba547"
server_key_iv: "4c8c89bdff8da548c1c69c1669aa677b"
server_key_version: "encryptionkey@0"
I20260810 18:01:29.409359 1310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:29.414076 1310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:29.414713 1310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:29.414911 1310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:29.422139 1310 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:41449
I20260810 18:01:29.422174 1378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:41449 every 8 connection(s)
I20260810 18:01:29.423202 1310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:29.423967 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 1310
I20260810 18:01:29.424122 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:29.424429 29005 external_mini_cluster.cc:1428] Setting key 13e44de7f86dae0c09e10e71adb18f6d
I20260810 18:01:29.426808 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:29.430773 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.425902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.28.83.126:41449)
0810 18:01:29.427188 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.427204 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.427241 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:01:29.427905 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.427922 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.428263 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.428611 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:29.429489 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:29.429494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:29.429519 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:29.429529 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:29.429608 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:29.429625 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:29.429777 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.429850 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.430103 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":683,"threads_started":1}
I20260810 18:01:29.432857 1379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f: Bootstrap starting.
I20260810 18:01:29.435323 1379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:29.436095 1379 log.cc:824] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:29.438313 1379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f: No bootstrap required, opened a new log
I20260810 18:01:29.442031 1379 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb89122517b44c63ad38f68362c13c6f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 41449 } }
I20260810 18:01:29.442274 1379 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:29.442355 1379 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb89122517b44c63ad38f68362c13c6f, State: Initialized, Role: FOLLOWER
I20260810 18:01:29.442818 1379 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb89122517b44c63ad38f68362c13c6f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 41449 } }
I20260810 18:01:29.442941 1379 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:29.443024 1379 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:29.443150 1379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:29.444128 1379 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb89122517b44c63ad38f68362c13c6f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 41449 } }
I20260810 18:01:29.444468 1379 leader_election.cc:302] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb89122517b44c63ad38f68362c13c6f; no voters:
I20260810 18:01:29.444783 1379 leader_election.cc:288] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:29.444928 1384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:29.445205 1384 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [term 1 LEADER]: Becoming Leader. State: Replica: bb89122517b44c63ad38f68362c13c6f, State: Running, Role: LEADER
I20260810 18:01:29.445535 1384 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb89122517b44c63ad38f68362c13c6f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 41449 } }
I20260810 18:01:29.446147 1379 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:29.447142 1385 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb89122517b44c63ad38f68362c13c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb89122517b44c63ad38f68362c13c6f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 41449 } } }
I20260810 18:01:29.447290 1385 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:29.447703 1393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:29.447655 1386 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb89122517b44c63ad38f68362c13c6f. Latest consensus state: current_term: 1 leader_uuid: "bb89122517b44c63ad38f68362c13c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb89122517b44c63ad38f68362c13c6f" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 41449 } } }
I20260810 18:01:29.447805 1386 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:29.450529 1393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:29.456514 1393 catalog_manager.cc:1347] Generated new cluster ID: 66c8ee2603b24135a17eb28663ef161d
I20260810 18:01:29.456604 1393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:29.474177 1393 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:29.474578 1393 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:29.483687 1393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f: Generated new TSK 0
I20260810 18:01:29.484520 1393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:01:29.496348 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:41449
--builtin_ntp_servers=127.28.83.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:01:29.617820 1404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:29.618098 1404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:29.618171 1404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:29.622229 1404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:29.622290 1404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:29.622396 1404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:29.627682 1404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:41449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:29.628865 1404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:29.630295 1404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:29.640156 1410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:29.640211 1409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:29.640308 1404 server_base.cc:1034] running on GCE node
W20260810 18:01:29.640151 1412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:29.640877 1404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:29.641655 1404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:29.642828 1404 hybrid_clock.cc:648] HybridClock initialized: now 1786384889642794 us; error 50 us; skew 500 ppm
I20260810 18:01:29.644820 1404 webserver.cc:460] Webserver started at http://127.28.83.65:42153/ using document root <none> and password file <none>
I20260810 18:01:29.645493 1404 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:29.645566 1404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:29.645828 1404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:29.648231 1404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "fc3f821b19fb443089c28cd1d1f7b20b"
format_stamp: "Formatted at 2026-08-10 18:01:29 on dist-test-slave-srgv"
server_key: "8bd05f2045428a1fd86a09d7de194b1e"
server_key_iv: "0d23b18698583f34e243199621e84cdd"
server_key_version: "encryptionkey@0"
I20260810 18:01:29.648880 1404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "fc3f821b19fb443089c28cd1d1f7b20b"
format_stamp: "Formatted at 2026-08-10 18:01:29 on dist-test-slave-srgv"
server_key: "8bd05f2045428a1fd86a09d7de194b1e"
server_key_iv: "0d23b18698583f34e243199621e84cdd"
server_key_version: "encryptionkey@0"
I20260810 18:01:29.652792 1404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:01:29.655496 1418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:29.656778 1404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:29.656910 1404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "fc3f821b19fb443089c28cd1d1f7b20b"
format_stamp: "Formatted at 2026-08-10 18:01:29 on dist-test-slave-srgv"
server_key: "8bd05f2045428a1fd86a09d7de194b1e"
server_key_iv: "0d23b18698583f34e243199621e84cdd"
server_key_version: "encryptionkey@0"
I20260810 18:01:29.657034 1404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:29.661942 1404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:29.662595 1404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:29.662791 1404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:29.663367 1404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:29.664342 1404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:29.664391 1404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:29.664462 1404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:29.664494 1404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:29.673578 1404 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:45843
I20260810 18:01:29.673599 1535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:45843 every 8 connection(s)
I20260810 18:01:29.674567 1404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:29.680902 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.676467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:34611 (local address 127.28.83.126:41449)
0810 18:01:29.676608 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.676612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.677724 (+ 1112us) server_negotiation.cc:407] Connection header received
0810 18:01:29.679152 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.679156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.679205 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.679296 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:29.680405 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:29.680409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:29.680411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:29.680424 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:29.680453 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:29.680462 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:29.680563 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.680712 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.680761 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:01:29.681845 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.676805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:41449 (local address 127.28.83.65:34611)
0810 18:01:29.677569 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:29.677614 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:29.678929 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:29.679477 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:29.679484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:29.679572 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:29.680130 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:01:29.680131 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:29.680181 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:01:29.680251 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:29.680553 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:29.680562 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:29.680568 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:01:29.680755 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 18:01:29.681007 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260810 18:01:29.683058 1536 heartbeater.cc:347] Connected to a master server at 127.28.83.126:41449
I20260810 18:01:29.683305 1536 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:29.683739 1536 heartbeater.cc:511] Master 127.28.83.126:41449 requested a full tablet report, sending...
I20260810 18:01:29.684293 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 1404
I20260810 18:01:29.684419 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:29.684805 29005 external_mini_cluster.cc:1428] Setting key a1fa750a6f68a035f24023fdf4336134
I20260810 18:01:29.685396 1343 ts_manager.cc:194] Registered new tserver with Master: fc3f821b19fb443089c28cd1d1f7b20b (127.28.83.65:45843)
I20260810 18:01:29.687031 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:41449
--builtin_ntp_servers=127.28.83.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:01:29.800133 1539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:29.800406 1539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:29.800447 1539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:29.804651 1539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:29.804724 1539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:29.804816 1539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:29.809715 1539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:41449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:29.810883 1539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:29.812287 1539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:29.821851 1547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:29.821859 1545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:29.821859 1544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:29.822139 1539 server_base.cc:1034] running on GCE node
I20260810 18:01:29.822741 1539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:29.823388 1539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:29.824577 1539 hybrid_clock.cc:648] HybridClock initialized: now 1786384889824567 us; error 30 us; skew 500 ppm
I20260810 18:01:29.826517 1539 webserver.cc:460] Webserver started at http://127.28.83.66:35501/ using document root <none> and password file <none>
I20260810 18:01:29.827100 1539 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:29.827208 1539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:29.827473 1539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:29.829597 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "93338cfd40924ba9b4547eabaa9c37ef"
format_stamp: "Formatted at 2026-08-10 18:01:29 on dist-test-slave-srgv"
server_key: "08abc338c151703ad1900da552e4bb69"
server_key_iv: "ba4ee0f97a9115be477e98130e23c675"
server_key_version: "encryptionkey@0"
I20260810 18:01:29.830169 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "93338cfd40924ba9b4547eabaa9c37ef"
format_stamp: "Formatted at 2026-08-10 18:01:29 on dist-test-slave-srgv"
server_key: "08abc338c151703ad1900da552e4bb69"
server_key_iv: "ba4ee0f97a9115be477e98130e23c675"
server_key_version: "encryptionkey@0"
I20260810 18:01:29.834014 1539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:29.836778 1553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:29.838076 1539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:29.838225 1539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "93338cfd40924ba9b4547eabaa9c37ef"
format_stamp: "Formatted at 2026-08-10 18:01:29 on dist-test-slave-srgv"
server_key: "08abc338c151703ad1900da552e4bb69"
server_key_iv: "ba4ee0f97a9115be477e98130e23c675"
server_key_version: "encryptionkey@0"
I20260810 18:01:29.838357 1539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:29.843084 1539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:29.843719 1539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:29.843907 1539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:29.844496 1539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:29.845547 1539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:29.845631 1539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:29.845697 1539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:29.845750 1539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:29.855005 1539 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:43901
I20260810 18:01:29.855028 1675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:43901 every 8 connection(s)
I20260810 18:01:29.855953 1539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:29.862529 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.858203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:35291 (local address 127.28.83.126:41449)
0810 18:01:29.858348 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:29.858351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:29.859384 (+ 1033us) server_negotiation.cc:407] Connection header received
0810 18:01:29.860797 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:29.860800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:29.860841 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:29.860920 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:29.862123 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:29.862126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:29.862128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:29.862141 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:29.862161 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:29.862170 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:29.862255 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:29.862376 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 18:01:29.862416 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:01:29.863497 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 1539
I20260810 18:01:29.863584 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:29.858547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:41449 (local address 127.28.83.66:35291)
0810 18:01:29.859228 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:29.859271 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:29.860601 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:29.861092 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:29.861099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:29.861187 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:29.861849 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:01:29.861851 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:29.861892 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:01:29.861963 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:29.862248 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:29.862255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:29.862259 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:01:29.862396 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 18:01:29.862630 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":130,"threads_started":1}
I20260810 18:01:29.863626 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:29.863963 29005 external_mini_cluster.cc:1428] Setting key 2281e912eb7b5a10fbba278f78ce9143
I20260810 18:01:29.864885 1676 heartbeater.cc:347] Connected to a master server at 127.28.83.126:41449
I20260810 18:01:29.865142 1676 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:29.865705 1676 heartbeater.cc:511] Master 127.28.83.126:41449 requested a full tablet report, sending...
I20260810 18:01:29.866171 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:41449
--builtin_ntp_servers=127.28.83.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 18:01:29.866837 1343 ts_manager.cc:194] Registered new tserver with Master: 93338cfd40924ba9b4547eabaa9c37ef (127.28.83.66:43901)
W20260810 18:01:29.982231 1679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:29.982532 1679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:29.982591 1679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:29.986605 1679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:29.986661 1679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:29.986717 1679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:29.991631 1679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:41449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:29.992735 1679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:29.994163 1679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:30.003710 1684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:30.003722 1687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:30.003846 1679 server_base.cc:1034] running on GCE node
W20260810 18:01:30.003736 1685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:30.004495 1679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:30.005100 1679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:30.006315 1679 hybrid_clock.cc:648] HybridClock initialized: now 1786384890006298 us; error 27 us; skew 500 ppm
I20260810 18:01:30.008296 1679 webserver.cc:460] Webserver started at http://127.28.83.67:36447/ using document root <none> and password file <none>
I20260810 18:01:30.008913 1679 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:30.009011 1679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:30.009276 1679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:30.011389 1679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "2b7180f684224af584b61841d209cfae"
format_stamp: "Formatted at 2026-08-10 18:01:30 on dist-test-slave-srgv"
server_key: "e645585ee0992e638f67a477a28a4070"
server_key_iv: "080857ff55d77ac6856538e25751b260"
server_key_version: "encryptionkey@0"
I20260810 18:01:30.011958 1679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "2b7180f684224af584b61841d209cfae"
format_stamp: "Formatted at 2026-08-10 18:01:30 on dist-test-slave-srgv"
server_key: "e645585ee0992e638f67a477a28a4070"
server_key_iv: "080857ff55d77ac6856538e25751b260"
server_key_version: "encryptionkey@0"
I20260810 18:01:30.015791 1679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:01:30.018594 1693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:30.019806 1679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:30.019930 1679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "2b7180f684224af584b61841d209cfae"
format_stamp: "Formatted at 2026-08-10 18:01:30 on dist-test-slave-srgv"
server_key: "e645585ee0992e638f67a477a28a4070"
server_key_iv: "080857ff55d77ac6856538e25751b260"
server_key_version: "encryptionkey@0"
I20260810 18:01:30.020048 1679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:30.024629 1679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:30.025238 1679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:30.025465 1679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:30.026021 1679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:30.026938 1679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:30.026993 1679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:30.027061 1679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:30.027099 1679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:30.035528 1679 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:45273
I20260810 18:01:30.035625 1812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:45273 every 8 connection(s)
I20260810 18:01:30.036481 1679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:30.037014 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 1679
I20260810 18:01:30.037151 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:30.037500 29005 external_mini_cluster.cc:1428] Setting key cc6f7274cab30449a54d8e5d88a06a5a
I20260810 18:01:30.043547 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.038809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:57645 (local address 127.28.83.126:41449)
0810 18:01:30.038969 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.038975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.040308 (+ 1333us) server_negotiation.cc:407] Connection header received
0810 18:01:30.041758 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.041761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.041806 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.041890 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.043040 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.043043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.043046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.043058 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.043079 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.043089 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.043197 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.043337 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.043384 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:01:30.044456 1815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.039319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:41449 (local address 127.28.83.67:57645)
0810 18:01:30.040159 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:30.040216 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:30.041552 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:30.042075 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:30.042082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:30.042171 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:30.042777 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:01:30.042779 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:30.042820 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:01:30.042887 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:30.043180 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:30.043189 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:30.043195 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:01:30.043373 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 18:01:30.043650 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":121,"threads_started":1}
I20260810 18:01:30.045667 1813 heartbeater.cc:347] Connected to a master server at 127.28.83.126:41449
I20260810 18:01:30.045950 1813 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:30.046386 1813 heartbeater.cc:511] Master 127.28.83.126:41449 requested a full tablet report, sending...
I20260810 18:01:30.047515 1343 ts_manager.cc:194] Registered new tserver with Master: 2b7180f684224af584b61841d209cfae (127.28.83.67:45273)
I20260810 18:01:30.051942 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:01:30.055538 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.053923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.28.83.126:41449)
0810 18:01:30.054067 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.054072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.054228 (+ 156us) server_negotiation.cc:407] Connection header received
0810 18:01:30.054330 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.054333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.054367 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.054436 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.055185 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.055187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.055189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.055198 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.055216 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.055224 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.055293 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.055340 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.055373 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:01:30.058610 1343 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:30.061784 1343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:30.074265 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.070628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:45843 (local address 127.0.0.1:60864)
0810 18:01:30.071279 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:30.071798 (+ 519us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:30.071887 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:30.072435 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:30.072441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:30.072516 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:30.073260 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:01:30.073264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:30.073290 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:01:30.073371 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:30.073867 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:30.073878 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:30.073884 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:01:30.073993 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 18:01:30.074071 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"spinlock_wait_cycles":5632,"thread_start_us":104,"threads_started":1}
I20260810 18:01:30.074409 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.071559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.28.83.66:43901)
0810 18:01:30.072136 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.072147 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.072169 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:01:30.072223 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.072227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.072399 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.072813 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.073679 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.073684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.073694 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.073719 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.073757 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.073788 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.074130 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.074179 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.074256 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":89,"threads_started":1}
I20260810 18:01:30.074805 1827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.071390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:43901 (local address 127.0.0.1:49098)
0810 18:01:30.071774 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:30.071789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:30.071887 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:30.072556 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:30.072560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:30.072574 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:30.073516 (+ 942us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:01:30.073518 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:30.073528 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:01:30.073549 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:30.073897 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:30.073900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:30.073904 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:01:30.073983 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:01:30.074633 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":559,"spinlock_wait_cycles":4224,"thread_start_us":101,"threads_started":1}
I20260810 18:01:30.075040 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.070754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.28.83.65:45843)
0810 18:01:30.072052 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.072062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.072082 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:01:30.072126 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.072129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.072288 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.072441 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.073661 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.073667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.073679 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.073692 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.073734 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.073763 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.074737 (+ 974us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.074791 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.074849 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":162,"threads_started":1}
I20260810 18:01:30.075052 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.070733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:45273 (local address 127.0.0.1:36684)
0810 18:01:30.071715 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:30.071736 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:30.071837 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:30.073223 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:30.073226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:30.073241 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:30.073750 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:01:30.073752 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:30.073762 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:01:30.073786 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:30.074200 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:30.074203 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:30.074207 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:01:30.074275 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:01:30.074311 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":134,"threads_started":1}
I20260810 18:01:30.075891 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.071330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.28.83.67:45273)
0810 18:01:30.072133 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.072145 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.072169 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:01:30.072223 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.072227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.072399 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.072534 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.073954 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.073964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.073975 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.073988 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.074031 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.074067 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.074160 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.074824 (+ 664us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.075724 (+ 900us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":157,"threads_started":1}
I20260810 18:01:30.077881 1605 tablet_service.cc:1467] Processing CreateTablet for tablet 40b8d17bf5134a26a0d4c6bc505cb22b (DEFAULT_TABLE table=test-table [id=add3a0a3c6054f1895b477bd48852877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:30.077878 1468 tablet_service.cc:1467] Processing CreateTablet for tablet 40b8d17bf5134a26a0d4c6bc505cb22b (DEFAULT_TABLE table=test-table [id=add3a0a3c6054f1895b477bd48852877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:30.078462 1746 tablet_service.cc:1467] Processing CreateTablet for tablet 40b8d17bf5134a26a0d4c6bc505cb22b (DEFAULT_TABLE table=test-table [id=add3a0a3c6054f1895b477bd48852877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:30.078992 1605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b8d17bf5134a26a0d4c6bc505cb22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:30.079073 1468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b8d17bf5134a26a0d4c6bc505cb22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:30.079435 1746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b8d17bf5134a26a0d4c6bc505cb22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:30.085670 1830 tablet_bootstrap.cc:492] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: Bootstrap starting.
I20260810 18:01:30.087008 1831 tablet_bootstrap.cc:492] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b: Bootstrap starting.
I20260810 18:01:30.088622 1830 tablet_bootstrap.cc:654] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:30.088796 1831 tablet_bootstrap.cc:654] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:30.089185 1832 tablet_bootstrap.cc:492] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: Bootstrap starting.
I20260810 18:01:30.089591 1831 log.cc:824] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:30.089593 1830 log.cc:824] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:30.091298 1832 tablet_bootstrap.cc:654] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:30.091621 1831 tablet_bootstrap.cc:492] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b: No bootstrap required, opened a new log
I20260810 18:01:30.091837 1831 ts_tablet_manager.cc:1397] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:01:30.092052 1830 tablet_bootstrap.cc:492] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: No bootstrap required, opened a new log
I20260810 18:01:30.092168 1832 log.cc:824] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:30.092260 1830 ts_tablet_manager.cc:1397] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260810 18:01:30.094895 1832 tablet_bootstrap.cc:492] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: No bootstrap required, opened a new log
I20260810 18:01:30.095132 1832 ts_tablet_manager.cc:1397] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:01:30.096133 1830 raft_consensus.cc:348] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } }
I20260810 18:01:30.096398 1830 raft_consensus.cc:374] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:30.096460 1830 raft_consensus.cc:729] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93338cfd40924ba9b4547eabaa9c37ef, State: Initialized, Role: FOLLOWER
I20260810 18:01:30.096519 1831 raft_consensus.cc:348] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } }
I20260810 18:01:30.096778 1831 raft_consensus.cc:374] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:30.096832 1831 raft_consensus.cc:729] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3f821b19fb443089c28cd1d1f7b20b, State: Initialized, Role: FOLLOWER
I20260810 18:01:30.096959 1830 consensus_queue.cc:260] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } }
I20260810 18:01:30.097239 1831 consensus_queue.cc:260] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } }
I20260810 18:01:30.097801 1676 heartbeater.cc:503] Master 127.28.83.126:41449 was elected leader, sending a full tablet report...
I20260810 18:01:30.098061 1830 ts_tablet_manager.cc:1428] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260810 18:01:30.098279 1831 ts_tablet_manager.cc:1428] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:01:30.098634 1536 heartbeater.cc:503] Master 127.28.83.126:41449 was elected leader, sending a full tablet report...
I20260810 18:01:30.099908 1832 raft_consensus.cc:348] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } }
I20260810 18:01:30.100173 1832 raft_consensus.cc:374] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:30.100260 1832 raft_consensus.cc:729] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7180f684224af584b61841d209cfae, State: Initialized, Role: FOLLOWER
I20260810 18:01:30.100697 1832 consensus_queue.cc:260] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } }
I20260810 18:01:30.101400 1813 heartbeater.cc:503] Master 127.28.83.126:41449 was elected leader, sending a full tablet report...
I20260810 18:01:30.101830 1832 ts_tablet_manager.cc:1428] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260810 18:01:30.106947 1677 tablet.cc:2369] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:30.133219 1836 raft_consensus.cc:482] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:30.133467 1836 raft_consensus.cc:504] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } }
I20260810 18:01:30.134670 1836 leader_election.cc:288] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc3f821b19fb443089c28cd1d1f7b20b (127.28.83.65:45843), 2b7180f684224af584b61841d209cfae (127.28.83.67:45273)
I20260810 18:01:30.136595 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.135063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:53417 (local address 127.28.83.67:45273)
0810 18:01:30.135248 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.135253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.135268 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:01:30.135317 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.135320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.135366 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.135455 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.136208 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.136211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.136214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.136228 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.136250 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.136262 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.136351 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.136431 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.136468 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 18:01:30.136595 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.134905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:45273 (local address 127.28.83.66:53417)
0810 18:01:30.135107 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:30.135125 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:30.135205 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:30.135491 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:30.135497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:30.135512 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:30.136043 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:01:30.136045 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:30.136055 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:01:30.136078 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:30.136342 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:30.136345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:30.136348 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:01:30.136423 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:01:30.136463 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 18:01:30.136826 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.135024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:47941 (local address 127.28.83.65:45843)
0810 18:01:30.135160 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.135164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.135446 (+ 282us) server_negotiation.cc:407] Connection header received
0810 18:01:30.135518 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.135521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.135563 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.135655 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.136442 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.136445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.136448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.136459 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.136487 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.136496 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.136592 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.136661 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.136700 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:01:30.136855 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.134912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:45843 (local address 127.28.83.66:47941)
0810 18:01:30.135317 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:30.135330 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:30.135433 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:30.135712 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:30.135717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:30.135736 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:30.136260 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:01:30.136262 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:30.136274 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:01:30.136307 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:30.136577 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:30.136580 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:30.136583 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:01:30.136662 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:01:30.136706 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":32,"thread_start_us":167,"threads_started":1}
I20260810 18:01:30.137344 1766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b8d17bf5134a26a0d4c6bc505cb22b" candidate_uuid: "93338cfd40924ba9b4547eabaa9c37ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7180f684224af584b61841d209cfae" is_pre_election: true
I20260810 18:01:30.137529 1488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b8d17bf5134a26a0d4c6bc505cb22b" candidate_uuid: "93338cfd40924ba9b4547eabaa9c37ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" is_pre_election: true
I20260810 18:01:30.137773 1766 raft_consensus.cc:2454] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93338cfd40924ba9b4547eabaa9c37ef in term 0.
I20260810 18:01:30.137919 1488 raft_consensus.cc:2454] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93338cfd40924ba9b4547eabaa9c37ef in term 0.
I20260810 18:01:30.138340 1554 leader_election.cc:302] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7180f684224af584b61841d209cfae, 93338cfd40924ba9b4547eabaa9c37ef; no voters:
I20260810 18:01:30.138653 1836 raft_consensus.cc:2781] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:30.138756 1836 raft_consensus.cc:482] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:30.138813 1836 raft_consensus.cc:3037] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:30.139878 1836 raft_consensus.cc:504] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } }
I20260810 18:01:30.140280 1836 leader_election.cc:288] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [CANDIDATE]: Term 1 election: Requested vote from peers fc3f821b19fb443089c28cd1d1f7b20b (127.28.83.65:45843), 2b7180f684224af584b61841d209cfae (127.28.83.67:45273)
I20260810 18:01:30.140645 1488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b8d17bf5134a26a0d4c6bc505cb22b" candidate_uuid: "93338cfd40924ba9b4547eabaa9c37ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc3f821b19fb443089c28cd1d1f7b20b"
I20260810 18:01:30.140792 1488 raft_consensus.cc:3037] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:30.141969 1488 raft_consensus.cc:2454] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93338cfd40924ba9b4547eabaa9c37ef in term 1.
I20260810 18:01:30.142244 1766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b8d17bf5134a26a0d4c6bc505cb22b" candidate_uuid: "93338cfd40924ba9b4547eabaa9c37ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b7180f684224af584b61841d209cfae"
I20260810 18:01:30.142287 1556 leader_election.cc:302] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93338cfd40924ba9b4547eabaa9c37ef, fc3f821b19fb443089c28cd1d1f7b20b; no voters:
I20260810 18:01:30.142369 1766 raft_consensus.cc:3037] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:30.142493 1836 raft_consensus.cc:2781] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:30.142755 1836 raft_consensus.cc:686] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 1 LEADER]: Becoming Leader. State: Replica: 93338cfd40924ba9b4547eabaa9c37ef, State: Running, Role: LEADER
I20260810 18:01:30.143090 1836 consensus_queue.cc:237] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } }
I20260810 18:01:30.143689 1766 raft_consensus.cc:2454] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93338cfd40924ba9b4547eabaa9c37ef in term 1.
I20260810 18:01:30.146705 1342 catalog_manager.cc:5495] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 93338cfd40924ba9b4547eabaa9c37ef (127.28.83.66). New cstate: current_term: 1 leader_uuid: "93338cfd40924ba9b4547eabaa9c37ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93338cfd40924ba9b4547eabaa9c37ef" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 43901 } health_report { overall_health: HEALTHY } } }
I20260810 18:01:30.174592 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.173120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.28.83.66:43901)
0810 18:01:30.173265 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.173269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.173282 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:01:30.173360 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.173362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.173488 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.173560 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.174284 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.174286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.174288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.174296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.174311 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.174318 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.174390 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.174453 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.174486 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 18:01:30.175905 1537 tablet.cc:2369] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:30.181118 1766 raft_consensus.cc:1261] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [term 1 FOLLOWER]: Refusing update from remote peer 93338cfd40924ba9b4547eabaa9c37ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:30.181161 1488 raft_consensus.cc:1261] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [term 1 FOLLOWER]: Refusing update from remote peer 93338cfd40924ba9b4547eabaa9c37ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:30.181985 1840 consensus_queue.cc:1035] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b7180f684224af584b61841d209cfae" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:30.182284 1836 consensus_queue.cc:1035] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc3f821b19fb443089c28cd1d1f7b20b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:30.190914 1845 mvcc.cc:205] Tried to move back new op lower bound from 7317032510173270016 to 7317032510028959744. Current Snapshot: MvccSnapshot[applied={T|T < 7317032510173270016}]
I20260810 18:01:30.198987 1343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:30.199146 1343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:30.201799 1343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb89122517b44c63ad38f68362c13c6f: Sending DeleteTablet for 3 replicas of tablet 40b8d17bf5134a26a0d4c6bc505cb22b
I20260810 18:01:30.202636 1746 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b8d17bf5134a26a0d4c6bc505cb22b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:30 UTC) from {username='slave'} at 127.0.0.1:36684
I20260810 18:01:30.202631 1605 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b8d17bf5134a26a0d4c6bc505cb22b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:30 UTC) from {username='slave'} at 127.0.0.1:49098
I20260810 18:01:30.203183 1853 tablet_replica.cc:330] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: stopping tablet replica
I20260810 18:01:30.203176 1468 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b8d17bf5134a26a0d4c6bc505cb22b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:30 UTC) from {username='slave'} at 127.0.0.1:60864
I20260810 18:01:30.203476 1852 tablet_replica.cc:330] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: stopping tablet replica
I20260810 18:01:30.203613 1854 tablet_replica.cc:330] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b: stopping tablet replica
I20260810 18:01:30.203939 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 1404
I20260810 18:01:30.203943 1854 raft_consensus.cc:2229] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:30.204231 1852 raft_consensus.cc:2229] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:30.204272 1854 raft_consensus.cc:2258] T 40b8d17bf5134a26a0d4c6bc505cb22b P fc3f821b19fb443089c28cd1d1f7b20b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:30.204726 1852 raft_consensus.cc:2258] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:30.203958 1853 raft_consensus.cc:2229] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:30.205163 1853 raft_consensus.cc:2258] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:30.207039 1853 ts_tablet_manager.cc:1905] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:30.207518 1852 ts_tablet_manager.cc:1905] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:30.210292 1852 ts_tablet_manager.cc:1918] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:30.210402 1852 log.cc:1196] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/40b8d17bf5134a26a0d4c6bc505cb22b
I20260810 18:01:30.210738 1852 ts_tablet_manager.cc:1939] T 40b8d17bf5134a26a0d4c6bc505cb22b P 93338cfd40924ba9b4547eabaa9c37ef: Deleting consensus metadata
I20260810 18:01:30.210760 1853 ts_tablet_manager.cc:1918] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:30.211001 1853 log.cc:1196] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/40b8d17bf5134a26a0d4c6bc505cb22b
I20260810 18:01:30.211376 1853 ts_tablet_manager.cc:1939] T 40b8d17bf5134a26a0d4c6bc505cb22b P 2b7180f684224af584b61841d209cfae: Deleting consensus metadata
I20260810 18:01:30.211928 1328 catalog_manager.cc:4841] TS 93338cfd40924ba9b4547eabaa9c37ef (127.28.83.66:43901): tablet 40b8d17bf5134a26a0d4c6bc505cb22b (table test-table [id=add3a0a3c6054f1895b477bd48852877]) successfully deleted
I20260810 18:01:30.212626 1326 catalog_manager.cc:4841] TS 2b7180f684224af584b61841d209cfae (127.28.83.67:45273): tablet 40b8d17bf5134a26a0d4c6bc505cb22b (table test-table [id=add3a0a3c6054f1895b477bd48852877]) successfully deleted
W20260810 18:01:30.214854 1328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:45843 (error 108)
I20260810 18:01:30.215773 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 1539
I20260810 18:01:30.215994 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.215645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:45843 (local address 127.0.0.1:60878)
0810 18:01:30.215803 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:30.215891 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 18:01:30.216544 1328 catalog_manager.cc:4584] TS fc3f821b19fb443089c28cd1d1f7b20b (127.28.83.65:45843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40b8d17bf5134a26a0d4c6bc505cb22b: Network error: Client connection negotiation failed: client connection to 127.28.83.65:45843: connect: Connection refused (error 111)
I20260810 18:01:30.222316 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 1679
I20260810 18:01:30.229074 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 1310
2026-08-10T18:01:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1003 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 18:01:30.252700 29005 mini_oidc.cc:152] Starting JWKS server
I20260810 18:01:30.254611 29005 webserver.cc:460] Webserver started at https://127.0.0.1:41411/ using document root <none> and password file <none>
I20260810 18:01:30.254815 29005 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:01:30.255191 29005 webserver.cc:460] Webserver started at http://127.0.0.1:35373/ using document root <none> and password file <none>
2026-08-10T18:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:30Z Disabled control of system clock
I20260810 18:01:30.265867 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:38065
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:40959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:38065
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41411/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/testchainca.cert with env {}
W20260810 18:01:30.379558 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:30.379851 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:30.379891 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:30.384238 1861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:30.384315 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:30.384330 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:30.384347 1861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:30.384392 1861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:01:30.384426 1861 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:01:30.384444 1861 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41411/jwks.json
I20260810 18:01:30.390043 1861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:40959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:38065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:38065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41411/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:30.391177 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:30.392218 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:30.400753 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:30.400768 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:30.400751 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:30.400934 1861 server_base.cc:1034] running on GCE node
I20260810 18:01:30.401541 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:30.402616 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:30.403816 1861 hybrid_clock.cc:648] HybridClock initialized: now 1786384890403807 us; error 31 us; skew 500 ppm
I20260810 18:01:30.405889 1861 webserver.cc:460] Webserver started at http://127.28.83.126:38519/ using document root <none> and password file <none>
I20260810 18:01:30.406479 1861 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:30.406579 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:30.406823 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:30.408908 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "352d32422bb8495c8d1b890e3ae57417"
format_stamp: "Formatted at 2026-08-10 18:01:30 on dist-test-slave-srgv"
server_key: "2fc45629d32443a3a7ed47d89d48d18b"
server_key_iv: "6037605390329ca0e69bf256da89d84d"
server_key_version: "encryptionkey@0"
I20260810 18:01:30.409559 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "352d32422bb8495c8d1b890e3ae57417"
format_stamp: "Formatted at 2026-08-10 18:01:30 on dist-test-slave-srgv"
server_key: "2fc45629d32443a3a7ed47d89d48d18b"
server_key_iv: "6037605390329ca0e69bf256da89d84d"
server_key_version: "encryptionkey@0"
I20260810 18:01:30.413213 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:30.415683 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:30.416878 1861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:30.417034 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "352d32422bb8495c8d1b890e3ae57417"
format_stamp: "Formatted at 2026-08-10 18:01:30 on dist-test-slave-srgv"
server_key: "2fc45629d32443a3a7ed47d89d48d18b"
server_key_iv: "6037605390329ca0e69bf256da89d84d"
server_key_version: "encryptionkey@0"
I20260810 18:01:30.417157 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:30.457341 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:30.458317 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:30.458544 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:30.465893 1861 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:38065
I20260810 18:01:30.465914 1928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:38065 every 8 connection(s)
I20260810 18:01:30.466886 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:30.470063 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:30.472929 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 1861
I20260810 18:01:30.473055 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:30.473368 29005 external_mini_cluster.cc:1428] Setting key 05ee7c03f90e69898dc76df2b762fba1
I20260810 18:01:30.476369 1930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417: Bootstrap starting.
I20260810 18:01:30.478977 1930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:30.479837 1930 log.cc:824] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:30.482241 1930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417: No bootstrap required, opened a new log
I20260810 18:01:30.483291 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.474805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.28.83.126:38065)
0810 18:01:30.475469 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.475484 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.475521 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:01:30.476138 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.476155 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.476593 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.476882 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.477225 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.478231 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.479481 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.480032 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.480159 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:30.481089 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.481095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.481111 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.481679 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.481752 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.481778 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.481884 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.482015 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.482296 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":160,"threads_started":1}
I20260810 18:01:30.486326 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:01:30.486724 1930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352d32422bb8495c8d1b890e3ae57417" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 38065 } }
I20260810 18:01:30.486969 1930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:30.487025 1930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352d32422bb8495c8d1b890e3ae57417, State: Initialized, Role: FOLLOWER
I20260810 18:01:30.487510 1930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352d32422bb8495c8d1b890e3ae57417" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 38065 } }
I20260810 18:01:30.487648 1930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:30.487732 1930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:30.487848 1930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:30.488849 1930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352d32422bb8495c8d1b890e3ae57417" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 38065 } }
I20260810 18:01:30.489192 1930 leader_election.cc:302] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 352d32422bb8495c8d1b890e3ae57417; no voters:
I20260810 18:01:30.489549 1930 leader_election.cc:288] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:30.489676 1935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:30.489900 1935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [term 1 LEADER]: Becoming Leader. State: Replica: 352d32422bb8495c8d1b890e3ae57417, State: Running, Role: LEADER
I20260810 18:01:30.490243 1935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352d32422bb8495c8d1b890e3ae57417" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 38065 } }
I20260810 18:01:30.490912 1930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:30.492041 1937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "352d32422bb8495c8d1b890e3ae57417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352d32422bb8495c8d1b890e3ae57417" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 38065 } } }
I20260810 18:01:30.492188 1937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:30.492023 1936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 352d32422bb8495c8d1b890e3ae57417. Latest consensus state: current_term: 1 leader_uuid: "352d32422bb8495c8d1b890e3ae57417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352d32422bb8495c8d1b890e3ae57417" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 38065 } } }
I20260810 18:01:30.492432 1936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:30.492700 1944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:30.496022 1944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:30.502122 1944 catalog_manager.cc:1347] Generated new cluster ID: 27201aed8c004b76a4976786a331b313
I20260810 18:01:30.502214 1944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:30.514428 1944 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:30.515390 1944 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:30.524569 1944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 352d32422bb8495c8d1b890e3ae57417: Generated new TSK 0
I20260810 18:01:30.525309 1944 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 18:01:30.568730 1960 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 18:01:30.568914 1960 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 18:01:30.569480 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.544506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.28.83.126:38065)
0810 18:01:30.544666 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.544671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.544900 (+ 229us) server_negotiation.cc:407] Connection header received
0810 18:01:30.545166 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.545168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.545225 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.545307 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:01:30.545649 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.546204 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.547419 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.547659 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.547739 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:30.548160 (+ 421us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:01:30.568910 (+ 20750us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 18:01:30.569290 (+ 380us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44776: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 18:01:30.569593 1933 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44776: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 18:01:30.571216 1959 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:30.544403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:38065 (local address 127.0.0.1:44776)
0810 18:01:30.544802 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:30.544819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:30.544963 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:30.545357 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:30.545360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:30.545369 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:01:30.545530 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:30.545538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.546323 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.546327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:30.547288 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:30.547299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.547784 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.547788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:30.547886 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:30.547943 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:01:30.569252 (+ 21309us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 18:01:30.569553 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.83.126:38065: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":128,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.83.126:38065: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 18:01:30.572099 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 1861
2026-08-10T18:01:30Z chronyd exiting
I20260810 18:01:30.610791 29005 test_util.cc:175] -----------------------------------------------
I20260810 18:01:30.610880 29005 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786384863806381-29005-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (359 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 18:01:30.611951 29005 mini_oidc.cc:152] Starting JWKS server
I20260810 18:01:30.612824 29005 webserver.cc:460] Webserver started at https://127.0.0.1:33849/ using document root <none> and password file <none>
I20260810 18:01:30.612905 29005 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:01:30.613150 29005 webserver.cc:460] Webserver started at http://127.0.0.1:37195/ using document root <none> and password file <none>
2026-08-10T18:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:30Z Disabled control of system clock
I20260810 18:01:30.622968 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44913
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:44913
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33849/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/testchainca.cert with env {}
W20260810 18:01:30.732781 1966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:30.733115 1966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:30.733222 1966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:30.737257 1966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:30.737372 1966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:30.737401 1966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:30.737452 1966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:30.737488 1966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:01:30.737521 1966 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:01:30.737555 1966 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33849/jwks.json
I20260810 18:01:30.743219 1966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:44913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33849/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:30.744379 1966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:30.745458 1966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:30.753984 1974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:30.753990 1972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:30.754000 1966 server_base.cc:1034] running on GCE node
W20260810 18:01:30.754009 1971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:30.754907 1966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:30.756011 1966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:30.757236 1966 hybrid_clock.cc:648] HybridClock initialized: now 1786384890757218 us; error 33 us; skew 500 ppm
I20260810 18:01:30.759363 1966 webserver.cc:460] Webserver started at http://127.28.83.126:43431/ using document root <none> and password file <none>
I20260810 18:01:30.759958 1966 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:30.760073 1966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:30.760303 1966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:30.762509 1966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "615cbdff835b41f9b033ec87fde9c066"
format_stamp: "Formatted at 2026-08-10 18:01:30 on dist-test-slave-srgv"
server_key: "582946190d58e3dba718259f5909a1c4"
server_key_iv: "9ca04d924a74cf19116b2d5778b5bc12"
server_key_version: "encryptionkey@0"
I20260810 18:01:30.763124 1966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "615cbdff835b41f9b033ec87fde9c066"
format_stamp: "Formatted at 2026-08-10 18:01:30 on dist-test-slave-srgv"
server_key: "582946190d58e3dba718259f5909a1c4"
server_key_iv: "9ca04d924a74cf19116b2d5778b5bc12"
server_key_version: "encryptionkey@0"
I20260810 18:01:30.766891 1966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:01:30.769564 1980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:30.770830 1966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:30.770989 1966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "615cbdff835b41f9b033ec87fde9c066"
format_stamp: "Formatted at 2026-08-10 18:01:30 on dist-test-slave-srgv"
server_key: "582946190d58e3dba718259f5909a1c4"
server_key_iv: "9ca04d924a74cf19116b2d5778b5bc12"
server_key_version: "encryptionkey@0"
I20260810 18:01:30.771139 1966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:30.791998 1966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:30.792747 1966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:30.792958 1966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:30.800105 1966 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:44913
I20260810 18:01:30.800117 2032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:44913 every 8 connection(s)
I20260810 18:01:30.801046 1966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:30.804286 2033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:30.809132 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 1966
I20260810 18:01:30.809226 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:30.809571 29005 external_mini_cluster.cc:1428] Setting key 72036c332772c9f18d320fb573238bee
I20260810 18:01:30.809850 2033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066: Bootstrap starting.
I20260810 18:01:30.812495 2033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:30.813373 2033 log.cc:824] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:30.815596 2033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066: No bootstrap required, opened a new log
I20260810 18:01:30.820191 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.810956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.28.83.126:44913)
0810 18:01:30.811629 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.811647 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.811688 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:01:30.812553 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.812598 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.813095 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.813512 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:30.813937 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.815298 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.816255 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.816919 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.817095 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:30.817928 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:30.817935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:30.817951 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:30.818463 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:30.818536 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:30.818563 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:30.818663 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:30.818843 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 18:01:30.819166 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260810 18:01:30.820303 2033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615cbdff835b41f9b033ec87fde9c066" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44913 } }
I20260810 18:01:30.820570 2033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:30.820636 2033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615cbdff835b41f9b033ec87fde9c066, State: Initialized, Role: FOLLOWER
I20260810 18:01:30.821102 2033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615cbdff835b41f9b033ec87fde9c066" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44913 } }
I20260810 18:01:30.821269 2033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:30.821341 2033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:30.821460 2033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:30.822554 2033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615cbdff835b41f9b033ec87fde9c066" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44913 } }
I20260810 18:01:30.822942 2033 leader_election.cc:302] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615cbdff835b41f9b033ec87fde9c066; no voters:
I20260810 18:01:30.823271 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:01:30.823297 2033 leader_election.cc:288] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:30.823428 2038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:30.823630 2038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [term 1 LEADER]: Becoming Leader. State: Replica: 615cbdff835b41f9b033ec87fde9c066, State: Running, Role: LEADER
I20260810 18:01:30.823957 2038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615cbdff835b41f9b033ec87fde9c066" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44913 } }
I20260810 18:01:30.824615 2033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:30.825212 2040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615cbdff835b41f9b033ec87fde9c066. Latest consensus state: current_term: 1 leader_uuid: "615cbdff835b41f9b033ec87fde9c066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615cbdff835b41f9b033ec87fde9c066" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44913 } } }
I20260810 18:01:30.825347 2040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:30.825704 2039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "615cbdff835b41f9b033ec87fde9c066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615cbdff835b41f9b033ec87fde9c066" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44913 } } }
I20260810 18:01:30.825802 2039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066 [sys.catalog]: This master's current role is: LEADER
W20260810 18:01:30.829622 2053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:01:30.829703 2053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:01:30.829787 2045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:30.830529 2045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:30.836237 2045 catalog_manager.cc:1347] Generated new cluster ID: ec526fb0411e489f8d70410a0c8e059b
I20260810 18:01:30.836316 2045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:30.854064 2045 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:30.855139 2045 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:30.868921 2045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 615cbdff835b41f9b033ec87fde9c066: Generated new TSK 0
I20260810 18:01:30.869880 2045 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 18:01:30.902813 2063 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 18:01:30.902925 2063 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 18:01:30.903697 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:30.883173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.28.83.126:44913)
0810 18:01:30.883309 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:30.883314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:30.883469 (+ 155us) server_negotiation.cc:407] Connection header received
0810 18:01:30.883629 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:30.883632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:30.883694 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:30.883819 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:01:30.884033 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.884914 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.885843 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.886190 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.886292 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:30.886579 (+ 287us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:01:30.903228 (+ 16649us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 18:01:30.903530 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49006: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 18:01:30.903771 2036 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49006: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 18:01:30.903779 2062 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:30.883081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44913 (local address 127.0.0.1:49006)
0810 18:01:30.883373 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:30.883385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:30.883484 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:30.883821 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:30.883823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:30.883829 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:01:30.883920 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:30.883925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.885034 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.885037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:30.885716 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:30.885725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:30.886312 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:30.886315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:30.886384 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:30.886407 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:01:30.903452 (+ 17045us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 18:01:30.903618 (+ 166us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.83.126:44913: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
I20260810 18:01:30.904459 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 1966
2026-08-10T18:01:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 18:01:30.944674 29005 mini_oidc.cc:152] Starting JWKS server
I20260810 18:01:30.945636 29005 webserver.cc:460] Webserver started at https://127.0.0.1:43441/ using document root <none> and password file <none>
I20260810 18:01:30.945725 29005 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:01:30.945928 29005 webserver.cc:460] Webserver started at http://127.0.0.1:37771/ using document root <none> and password file <none>
2026-08-10T18:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:30Z Disabled control of system clock
I20260810 18:01:30.955616 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33815
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:43927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:33815
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43441/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 18:01:31.066710 2069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:31.067026 2069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:31.067066 2069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:31.071295 2069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:31.071368 2069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:31.071384 2069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:31.071403 2069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:31.071419 2069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:01:31.071435 2069 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:01:31.071448 2069 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43441/jwks.json
I20260810 18:01:31.076543 2069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:43927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:33815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43441/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:31.077693 2069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:31.079092 2069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:31.087752 2075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:31.087725 2077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:31.087769 2074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:31.088021 2069 server_base.cc:1034] running on GCE node
I20260810 18:01:31.088936 2069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:31.090183 2069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:31.091379 2069 hybrid_clock.cc:648] HybridClock initialized: now 1786384891091348 us; error 49 us; skew 500 ppm
I20260810 18:01:31.093397 2069 webserver.cc:460] Webserver started at http://127.28.83.126:37193/ using document root <none> and password file <none>
I20260810 18:01:31.094074 2069 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:31.094169 2069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:31.094415 2069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:31.096539 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "23ff970d825c45f6a9800e953b13c106"
format_stamp: "Formatted at 2026-08-10 18:01:31 on dist-test-slave-srgv"
server_key: "f2fead685ecc584f1bd0ee1ef78ce017"
server_key_iv: "3c43b62e1317856fc69126e77b1c4f75"
server_key_version: "encryptionkey@0"
I20260810 18:01:31.097151 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "23ff970d825c45f6a9800e953b13c106"
format_stamp: "Formatted at 2026-08-10 18:01:31 on dist-test-slave-srgv"
server_key: "f2fead685ecc584f1bd0ee1ef78ce017"
server_key_iv: "3c43b62e1317856fc69126e77b1c4f75"
server_key_version: "encryptionkey@0"
I20260810 18:01:31.100917 2069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:31.103447 2083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:31.104717 2069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:31.104856 2069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "23ff970d825c45f6a9800e953b13c106"
format_stamp: "Formatted at 2026-08-10 18:01:31 on dist-test-slave-srgv"
server_key: "f2fead685ecc584f1bd0ee1ef78ce017"
server_key_iv: "3c43b62e1317856fc69126e77b1c4f75"
server_key_version: "encryptionkey@0"
I20260810 18:01:31.104972 2069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:31.130308 2069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:31.131073 2069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:31.131316 2069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:31.138653 2069 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:33815
I20260810 18:01:31.138689 2137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:33815 every 8 connection(s)
I20260810 18:01:31.139618 2069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:31.142089 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 2069
I20260810 18:01:31.142226 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:31.142532 29005 external_mini_cluster.cc:1428] Setting key d8d4874274e6726531fac434dda6ca3d
I20260810 18:01:31.143324 2138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:31.149588 2138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106: Bootstrap starting.
I20260810 18:01:31.151938 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:31.143924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.28.83.126:33815)
0810 18:01:31.144589 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:31.144605 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:31.144640 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:01:31.145290 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:31.145306 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:31.145796 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:31.146155 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:31.146643 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.147603 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.148544 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.149031 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.149214 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:31.150231 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:31.150236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:31.150249 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:31.150640 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:31.150708 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:31.150733 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:31.150840 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:31.150976 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 18:01:31.151235 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":144,"threads_started":1}
I20260810 18:01:31.151937 2138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:31.152822 2138 log.cc:824] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:31.154891 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:01:31.155225 2138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106: No bootstrap required, opened a new log
I20260810 18:01:31.159072 2138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ff970d825c45f6a9800e953b13c106" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33815 } }
I20260810 18:01:31.159363 2138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:31.159448 2138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ff970d825c45f6a9800e953b13c106, State: Initialized, Role: FOLLOWER
I20260810 18:01:31.159938 2138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ff970d825c45f6a9800e953b13c106" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33815 } }
I20260810 18:01:31.160096 2138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:31.160185 2138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:31.160286 2138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:31.161478 2138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ff970d825c45f6a9800e953b13c106" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33815 } }
I20260810 18:01:31.161857 2138 leader_election.cc:302] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23ff970d825c45f6a9800e953b13c106; no voters:
I20260810 18:01:31.162247 2138 leader_election.cc:288] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:31.162381 2143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:31.162632 2143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [term 1 LEADER]: Becoming Leader. State: Replica: 23ff970d825c45f6a9800e953b13c106, State: Running, Role: LEADER
I20260810 18:01:31.162997 2143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ff970d825c45f6a9800e953b13c106" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33815 } }
I20260810 18:01:31.163667 2138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:31.164986 2144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23ff970d825c45f6a9800e953b13c106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ff970d825c45f6a9800e953b13c106" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33815 } } }
I20260810 18:01:31.165155 2144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:31.165446 2145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23ff970d825c45f6a9800e953b13c106. Latest consensus state: current_term: 1 leader_uuid: "23ff970d825c45f6a9800e953b13c106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ff970d825c45f6a9800e953b13c106" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33815 } } }
I20260810 18:01:31.165568 2145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:31.165668 2152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:31.168615 2152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:31.174633 2152 catalog_manager.cc:1347] Generated new cluster ID: a170d520c1734a798afaed23151e5ea7
I20260810 18:01:31.174722 2152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:31.194968 2152 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:31.195992 2152 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:31.202030 2152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23ff970d825c45f6a9800e953b13c106: Generated new TSK 0
I20260810 18:01:31.202757 2152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:01:31.220480 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:31.215233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.28.83.126:33815)
0810 18:01:31.215379 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:31.215384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:31.215516 (+ 132us) server_negotiation.cc:407] Connection header received
0810 18:01:31.215649 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:31.215653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:31.215710 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:31.215839 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:01:31.216056 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.216907 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.217863 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.218201 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.218311 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:31.218600 (+ 289us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:01:31.220100 (+ 1500us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 18:01:31.220337 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 18:01:31.220568 2141 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 18:01:31.222155 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:31.215108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33815 (local address 127.0.0.1:43238)
0810 18:01:31.215428 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:31.215441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:31.215535 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:31.215842 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:31.215844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:31.215850 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:01:31.215949 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:31.215953 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.217047 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.217049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:31.217745 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:31.217754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.218321 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.218324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:31.218389 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:31.218411 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:01:31.220269 (+ 1858us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 18:01:31.221987 (+ 1718us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.83.126:33815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43441/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":1524,"thread_start_us":112,"threads_started":1}
I20260810 18:01:31.222506 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 2069
E20260810 18:01:31.231482 2168 webserver.cc:533] Webserver: sslize failed:
E20260810 18:01:31.231564 2168 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T18:01:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T18:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:31Z Disabled control of system clock
I20260810 18:01:31.272585 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:39199
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:39199
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/testchainca.cert with env {}
W20260810 18:01:31.388278 2172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:31.388566 2172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:31.388607 2172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:31.392781 2172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:31.392858 2172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:31.392874 2172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:31.392892 2172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:31.392907 2172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:01:31.392925 2172 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:01:31.392937 2172 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 18:01:31.398640 2172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:39199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:39199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:31.399767 2172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:31.400754 2172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:31.409382 2178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:31.409566 2172 server_base.cc:1034] running on GCE node
W20260810 18:01:31.409430 2180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:31.409471 2177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:31.410301 2172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:31.411427 2172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:31.412626 2172 hybrid_clock.cc:648] HybridClock initialized: now 1786384891412605 us; error 33 us; skew 500 ppm
I20260810 18:01:31.414621 2172 webserver.cc:460] Webserver started at http://127.28.83.126:33783/ using document root <none> and password file <none>
I20260810 18:01:31.415221 2172 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:31.415318 2172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:31.415555 2172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:31.417728 2172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "ca0be0b09d4c46ad8367a3d94f13055e"
format_stamp: "Formatted at 2026-08-10 18:01:31 on dist-test-slave-srgv"
server_key: "b66deb9e2241e0b366ca78a22fa2788a"
server_key_iv: "d899820b4998494019547aedce6b764f"
server_key_version: "encryptionkey@0"
I20260810 18:01:31.418329 2172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "ca0be0b09d4c46ad8367a3d94f13055e"
format_stamp: "Formatted at 2026-08-10 18:01:31 on dist-test-slave-srgv"
server_key: "b66deb9e2241e0b366ca78a22fa2788a"
server_key_iv: "d899820b4998494019547aedce6b764f"
server_key_version: "encryptionkey@0"
I20260810 18:01:31.422065 2172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:01:31.424464 2186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:31.425700 2172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:31.425838 2172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "ca0be0b09d4c46ad8367a3d94f13055e"
format_stamp: "Formatted at 2026-08-10 18:01:31 on dist-test-slave-srgv"
server_key: "b66deb9e2241e0b366ca78a22fa2788a"
server_key_iv: "d899820b4998494019547aedce6b764f"
server_key_version: "encryptionkey@0"
I20260810 18:01:31.425954 2172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:31.458834 2172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:31.459640 2172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:31.459873 2172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:31.467000 2172 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:39199
I20260810 18:01:31.467007 2238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:39199 every 8 connection(s)
I20260810 18:01:31.467995 2172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:31.469671 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 2172
I20260810 18:01:31.469794 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:31.470118 29005 external_mini_cluster.cc:1428] Setting key 9c47c1b4086bca994ce05288058852a0
I20260810 18:01:31.471325 2239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:31.477022 2239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e: Bootstrap starting.
I20260810 18:01:31.479417 2239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:31.479453 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:31.471568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.28.83.126:39199)
0810 18:01:31.472217 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:31.472232 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:31.472270 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:01:31.473008 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:31.473024 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:31.473481 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:31.473825 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:31.474262 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.475224 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.476115 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.476584 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.476795 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:31.477753 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:31.477758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:31.477771 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:01:31.478155 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:31.478213 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:01:31.478233 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:31.478343 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:31.478501 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 18:01:31.478763 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":145,"threads_started":1}
I20260810 18:01:31.480156 2239 log.cc:824] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:31.482391 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:01:31.482502 2239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e: No bootstrap required, opened a new log
I20260810 18:01:31.486387 2239 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0be0b09d4c46ad8367a3d94f13055e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39199 } }
I20260810 18:01:31.486639 2239 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:31.486711 2239 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca0be0b09d4c46ad8367a3d94f13055e, State: Initialized, Role: FOLLOWER
I20260810 18:01:31.487116 2239 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0be0b09d4c46ad8367a3d94f13055e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39199 } }
I20260810 18:01:31.487267 2239 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:31.487341 2239 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:31.487439 2239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:31.488425 2239 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0be0b09d4c46ad8367a3d94f13055e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39199 } }
I20260810 18:01:31.488799 2239 leader_election.cc:302] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca0be0b09d4c46ad8367a3d94f13055e; no voters:
I20260810 18:01:31.489172 2239 leader_election.cc:288] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:31.489326 2244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:31.489607 2244 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [term 1 LEADER]: Becoming Leader. State: Replica: ca0be0b09d4c46ad8367a3d94f13055e, State: Running, Role: LEADER
I20260810 18:01:31.489910 2244 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0be0b09d4c46ad8367a3d94f13055e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39199 } }
I20260810 18:01:31.490489 2239 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:31.491576 2246 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca0be0b09d4c46ad8367a3d94f13055e. Latest consensus state: current_term: 1 leader_uuid: "ca0be0b09d4c46ad8367a3d94f13055e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0be0b09d4c46ad8367a3d94f13055e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39199 } } }
I20260810 18:01:31.491755 2246 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:31.492060 2245 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca0be0b09d4c46ad8367a3d94f13055e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0be0b09d4c46ad8367a3d94f13055e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39199 } } }
I20260810 18:01:31.492151 2245 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:31.492270 2254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:31.494990 2254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:31.501273 2254 catalog_manager.cc:1347] Generated new cluster ID: 6dbe0b6d1de3409abb5589b66461b16a
I20260810 18:01:31.501358 2254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:31.508632 2254 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:31.509655 2254 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:31.513809 2254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca0be0b09d4c46ad8367a3d94f13055e: Generated new TSK 0
I20260810 18:01:31.514560 2254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:01:32.061710 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:31.523041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.28.83.126:39199)
0810 18:01:31.523187 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:31.523192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:31.523359 (+ 167us) server_negotiation.cc:407] Connection header received
0810 18:01:31.523517 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:31.523519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:31.523568 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:31.523669 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:01:31.523941 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.524474 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.525746 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.525965 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.526043 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:31.526431 (+ 388us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:01:32.061219 (+534788us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 18:01:32.061573 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47598: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 18:01:32.061795 2242 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47598: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 18:01:32.062003 2269 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:31.522945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39199 (local address 127.0.0.1:47598)
0810 18:01:31.523263 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:31.523278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:31.523398 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:31.523689 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:31.523693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:31.523708 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:01:31.523832 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:31.523839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.524607 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.524611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:31.525620 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:31.525631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:31.526095 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:31.526099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:31.526192 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:31.526223 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:01:32.061556 (+535333us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 18:01:32.061763 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.83.126:39199: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":122,"threads_started":1}
I20260810 18:01:32.062645 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 2172
2026-08-10T18:01:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (820 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:32 dist-test-slave-srgv krb5kdc[2282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:32 dist-test-slave-srgv krb5kdc[2282](info): set up 2 sockets
Aug 10 18:01:32 dist-test-slave-srgv krb5kdc[2282](info): commencing operation
krb5kdc: starting...
W20260810 18:01:33.368958 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:33 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:33.496778 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:45531
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:45531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:33.614648 2295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:33.614956 2295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:33.615025 2295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:33.619107 2295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:33.619199 2295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:33.619236 2295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:33.619297 2295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:33.619336 2295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:33.624595 2295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:45531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:45531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:33.625746 2295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:33.626778 2295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:01:33.635360 2295 server_base.cc:1034] running on GCE node
W20260810 18:01:33.635321 2300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:33.635356 2303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:33.635303 2301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:33.636333 2295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:33.637501 2295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:33.638712 2295 hybrid_clock.cc:648] HybridClock initialized: now 1786384893638706 us; error 33 us; skew 500 ppm
Aug 10 18:01:33 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:33.641739 2295 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:33.642700 2295 webserver.cc:460] Webserver started at http://127.28.83.126:34723/ using document root <none> and password file <none>
I20260810 18:01:33.643294 2295 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:33.643404 2295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:33.643646 2295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:33.645859 2295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "f1bf4f8282d3424590a6c82a58d6fb5e"
format_stamp: "Formatted at 2026-08-10 18:01:33 on dist-test-slave-srgv"
server_key: "9a515f438c2e374c36eb66fc881890af"
server_key_iv: "2462d727968b7af97ec8ffa9cdfc2642"
server_key_version: "encryptionkey@0"
I20260810 18:01:33.646456 2295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "f1bf4f8282d3424590a6c82a58d6fb5e"
format_stamp: "Formatted at 2026-08-10 18:01:33 on dist-test-slave-srgv"
server_key: "9a515f438c2e374c36eb66fc881890af"
server_key_iv: "2462d727968b7af97ec8ffa9cdfc2642"
server_key_version: "encryptionkey@0"
I20260810 18:01:33.650282 2295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:01:33.652673 2310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:33.653990 2295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:01:33.654134 2295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "f1bf4f8282d3424590a6c82a58d6fb5e"
format_stamp: "Formatted at 2026-08-10 18:01:33 on dist-test-slave-srgv"
server_key: "9a515f438c2e374c36eb66fc881890af"
server_key_iv: "2462d727968b7af97ec8ffa9cdfc2642"
server_key_version: "encryptionkey@0"
I20260810 18:01:33.654261 2295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:33.668533 2295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:33.672112 2295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:33.672351 2295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:33.679710 2295 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:45531
I20260810 18:01:33.679738 2363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:45531 every 8 connection(s)
I20260810 18:01:33.680763 2295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:33.683585 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:33.685232 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 2295
I20260810 18:01:33.685349 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:33.685678 29005 external_mini_cluster.cc:1428] Setting key b07b7569a6041d661cc14cd6a232ba85
I20260810 18:01:33.690541 2364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e: Bootstrap starting.
I20260810 18:01:33.693223 2364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:33.694125 2364 log.cc:824] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:01:33 dist-test-slave-srgv krb5kdc[2282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:33.696269 2364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e: No bootstrap required, opened a new log
I20260810 18:01:33.700970 2364 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bf4f8282d3424590a6c82a58d6fb5e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45531 } }
I20260810 18:01:33.701202 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:33.687214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.28.83.126:45531)
0810 18:01:33.687657 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:33.687677 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:33.687720 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:01:33.688600 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:33.688626 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:33.689039 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:33.689445 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:33.690180 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:33.691602 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:33.692441 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:33.693109 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:33.693297 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:33.696099 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:33.696104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:33.696120 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:33.696157 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:33.698143 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:33.698588 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:33.698597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:33.698605 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:33.698776 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:33.699041 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:33.699045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:33.699047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:33.699532 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:33.699737 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:33.699935 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 18:01:33.700248 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":124,"threads_started":1}
I20260810 18:01:33.701234 2364 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:33.701288 2364 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1bf4f8282d3424590a6c82a58d6fb5e, State: Initialized, Role: FOLLOWER
I20260810 18:01:33.701747 2364 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bf4f8282d3424590a6c82a58d6fb5e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45531 } }
I20260810 18:01:33.701869 2364 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:33.701954 2364 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:33.702075 2364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:33.703218 2364 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bf4f8282d3424590a6c82a58d6fb5e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45531 } }
I20260810 18:01:33.703569 2364 leader_election.cc:302] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1bf4f8282d3424590a6c82a58d6fb5e; no voters:
I20260810 18:01:33.703888 2364 leader_election.cc:288] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:33.704020 2369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:33.704268 2369 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [term 1 LEADER]: Becoming Leader. State: Replica: f1bf4f8282d3424590a6c82a58d6fb5e, State: Running, Role: LEADER
I20260810 18:01:33.704578 2369 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bf4f8282d3424590a6c82a58d6fb5e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45531 } }
I20260810 18:01:33.705101 2364 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:33.706351 2370 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1bf4f8282d3424590a6c82a58d6fb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bf4f8282d3424590a6c82a58d6fb5e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45531 } } }
I20260810 18:01:33.706503 2370 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:33.706352 2371 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1bf4f8282d3424590a6c82a58d6fb5e. Latest consensus state: current_term: 1 leader_uuid: "f1bf4f8282d3424590a6c82a58d6fb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bf4f8282d3424590a6c82a58d6fb5e" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45531 } } }
I20260810 18:01:33.706602 2371 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:33.706955 2378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:33.709803 2378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:33.715045 2378 catalog_manager.cc:1347] Generated new cluster ID: 44fa11e35a0d46459f96a5c12f05440a
I20260810 18:01:33.715149 2378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:33.738430 2378 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:33.739434 2378 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:33.748121 2378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1bf4f8282d3424590a6c82a58d6fb5e: Generated new TSK 0
I20260810 18:01:33.748883 2378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:33.787288 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:45531
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:33.907485 2392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:33.907786 2392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:33.907832 2392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:33.911829 2392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:33.911885 2392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:33.911940 2392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:33.917008 2392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:45531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:33.918113 2392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:33.919059 2392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:33.928607 2397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:33.928691 2398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:33.928718 2392 server_base.cc:1034] running on GCE node
W20260810 18:01:33.928607 2400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:33.929318 2392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:33.929986 2392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:33.931169 2392 hybrid_clock.cc:648] HybridClock initialized: now 1786384893931147 us; error 32 us; skew 500 ppm
Aug 10 18:01:33 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:33.934160 2392 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:33.935096 2392 webserver.cc:460] Webserver started at http://127.28.83.65:37645/ using document root <none> and password file <none>
I20260810 18:01:33.935665 2392 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:33.935731 2392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:33.935977 2392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:33.938298 2392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "fbfeedf4208844cf97ea53f89e54a0ed"
format_stamp: "Formatted at 2026-08-10 18:01:33 on dist-test-slave-srgv"
server_key: "ad30f5f252317d92ca0ddd3fa1c8a89e"
server_key_iv: "d3bfe4f4ad89e16cf786f14c2e78a93c"
server_key_version: "encryptionkey@0"
I20260810 18:01:33.938897 2392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "fbfeedf4208844cf97ea53f89e54a0ed"
format_stamp: "Formatted at 2026-08-10 18:01:33 on dist-test-slave-srgv"
server_key: "ad30f5f252317d92ca0ddd3fa1c8a89e"
server_key_iv: "d3bfe4f4ad89e16cf786f14c2e78a93c"
server_key_version: "encryptionkey@0"
I20260810 18:01:33.942762 2392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:01:33.947365 2408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:33.949343 2392 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260810 18:01:33.949532 2392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "fbfeedf4208844cf97ea53f89e54a0ed"
format_stamp: "Formatted at 2026-08-10 18:01:33 on dist-test-slave-srgv"
server_key: "ad30f5f252317d92ca0ddd3fa1c8a89e"
server_key_iv: "d3bfe4f4ad89e16cf786f14c2e78a93c"
server_key_version: "encryptionkey@0"
I20260810 18:01:33.949677 2392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:33.978048 2392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:33.982025 2392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:33.982199 2392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:33.982779 2392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:33.983608 2392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:33.983657 2392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:33.983698 2392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:33.983722 2392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:33.993119 2392 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:43097
I20260810 18:01:33.993211 2522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:43097 every 8 connection(s)
I20260810 18:01:33.994318 2392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:33.999305 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 2392
I20260810 18:01:33.999446 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:33.999706 29005 external_mini_cluster.cc:1428] Setting key 871adfd8781b57b8e027f7158be282b4
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:01:34.012354 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:33.996569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:52741 (local address 127.28.83.126:45531)
0810 18:01:33.996713 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:33.996716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:33.997923 (+ 1207us) server_negotiation.cc:407] Connection header received
0810 18:01:33.999697 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:33.999701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:33.999763 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:33.999843 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:34.001366 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.002212 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.004002 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.004431 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.004520 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:34.007848 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:34.007852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:34.007854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:34.007880 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:34.009891 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.010527 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.010531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.010532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.010631 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.011078 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.011085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.011089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.011343 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:34.011455 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:34.012029 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 18:01:34.012172 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:01:34.013477 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:33.997079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:45531 (local address 127.28.83.65:52741)
0810 18:01:33.997727 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:33.997790 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:33.999453 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:34.000022 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:34.000044 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:34.000537 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:34.001211 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:34.001234 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.002360 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.002363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:34.003838 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:34.003851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.004783 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.004786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:34.005027 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:34.005811 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:34.005835 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:34.007679 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:34.010015 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:34.010020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:34.010034 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:34.010369 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:34.010750 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:34.010756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:34.010760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:34.010951 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:34.011457 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:34.011465 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:34.011773 (+ 308us) client_negotiation.cc:769] Sending connection context
0810 18:01:34.011985 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 18:01:34.012315 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:34.014698 2524 heartbeater.cc:347] Connected to a master server at 127.28.83.126:45531
I20260810 18:01:34.014931 2524 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:34.015416 2524 heartbeater.cc:511] Master 127.28.83.126:45531 requested a full tablet report, sending...
I20260810 18:01:34.017158 2327 ts_manager.cc:194] Registered new tserver with Master: fbfeedf4208844cf97ea53f89e54a0ed (127.28.83.65:43097)
I20260810 18:01:34.018337 2327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:52741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:34.032614 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:45531
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:34.149660 2529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:34.149950 2529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:34.150029 2529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:34.154392 2529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:34.154462 2529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:34.154568 2529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:34.160197 2529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:45531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:34.161334 2529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:34.162458 2529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:34.172520 2535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:34.172600 2537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:34.172588 2534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:34.172906 2529 server_base.cc:1034] running on GCE node
I20260810 18:01:34.173316 2529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:34.173976 2529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:34.175173 2529 hybrid_clock.cc:648] HybridClock initialized: now 1786384894175153 us; error 34 us; skew 500 ppm
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:34.178077 2529 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:34.179039 2529 webserver.cc:460] Webserver started at http://127.28.83.66:44203/ using document root <none> and password file <none>
I20260810 18:01:34.179647 2529 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:34.179750 2529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:34.179991 2529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:34.182230 2529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "f5b02c82854e47f384fbce287eaac448"
format_stamp: "Formatted at 2026-08-10 18:01:34 on dist-test-slave-srgv"
server_key: "637d56e66a7b27d985c00c9c16a6a7ab"
server_key_iv: "09f6709e9f9af67eab2218d0dcc0abb9"
server_key_version: "encryptionkey@0"
I20260810 18:01:34.182839 2529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "f5b02c82854e47f384fbce287eaac448"
format_stamp: "Formatted at 2026-08-10 18:01:34 on dist-test-slave-srgv"
server_key: "637d56e66a7b27d985c00c9c16a6a7ab"
server_key_iv: "09f6709e9f9af67eab2218d0dcc0abb9"
server_key_version: "encryptionkey@0"
I20260810 18:01:34.186898 2529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:01:34.189559 2544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:34.190832 2529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:34.190980 2529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "f5b02c82854e47f384fbce287eaac448"
format_stamp: "Formatted at 2026-08-10 18:01:34 on dist-test-slave-srgv"
server_key: "637d56e66a7b27d985c00c9c16a6a7ab"
server_key_iv: "09f6709e9f9af67eab2218d0dcc0abb9"
server_key_version: "encryptionkey@0"
I20260810 18:01:34.191107 2529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:34.211674 2529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:34.215472 2529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:34.215725 2529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:34.216349 2529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:34.217219 2529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:34.217294 2529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:34.217365 2529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:34.217450 2529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 18:01:34.226279 2529 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:35009
I20260810 18:01:34.226312 2658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:35009 every 8 connection(s)
I20260810 18:01:34.227185 2529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:34.229810 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 2529
I20260810 18:01:34.229936 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:34.230281 29005 external_mini_cluster.cc:1428] Setting key 49577ccc40510df3afea26b63c8c8d81
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
I20260810 18:01:34.243144 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:34.228960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:45605 (local address 127.28.83.126:45531)
0810 18:01:34.229173 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:34.229178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:34.230109 (+ 931us) server_negotiation.cc:407] Connection header received
0810 18:01:34.231782 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:34.231785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:34.231839 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:34.231946 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:34.233206 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.234122 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.235349 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.235767 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.235898 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:34.239163 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:34.239166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:34.239168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:34.239194 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:34.240710 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.241361 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.241365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.241366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.241518 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.241932 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.241938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.241943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.242170 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:34.242289 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:34.242901 (+ 612us) server_negotiation.cc:299] Negotiation successful
0810 18:01:34.243016 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:01:34.244268 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:34.229342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:45531 (local address 127.28.83.66:45605)
0810 18:01:34.229924 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:34.229988 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:34.231555 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:34.232117 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:34.232124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:34.232579 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:34.233055 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:34.233068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.234283 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.234286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:34.235181 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:34.235191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.235995 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.236001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:34.236264 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:34.237104 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:34.237120 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:34.238961 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:34.240853 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:34.240860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:34.240878 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:34.241250 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:34.241621 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:34.241624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:34.241627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:34.241796 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:34.242286 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:34.242294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:34.242579 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 18:01:34.242836 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 18:01:34.243166 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":125,"threads_started":1}
I20260810 18:01:34.245690 2659 heartbeater.cc:347] Connected to a master server at 127.28.83.126:45531
I20260810 18:01:34.245990 2659 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:34.246645 2659 heartbeater.cc:511] Master 127.28.83.126:45531 requested a full tablet report, sending...
I20260810 18:01:34.247803 2327 ts_manager.cc:194] Registered new tserver with Master: f5b02c82854e47f384fbce287eaac448 (127.28.83.66:35009)
I20260810 18:01:34.248536 2327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:45605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:01:34.261354 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:45531
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:34.376921 2664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:34.377224 2664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:34.377270 2664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:34.381672 2664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:34.381747 2664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:34.381839 2664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:34.387341 2664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:45531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:34.388477 2664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:34.389616 2664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:34.399353 2672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:34.399348 2670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:34.399353 2669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:34.399981 2664 server_base.cc:1034] running on GCE node
I20260810 18:01:34.400420 2664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:34.401069 2664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:34.402256 2664 hybrid_clock.cc:648] HybridClock initialized: now 1786384894402246 us; error 31 us; skew 500 ppm
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:34.405164 2664 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:34.406200 2664 webserver.cc:460] Webserver started at http://127.28.83.67:37649/ using document root <none> and password file <none>
I20260810 18:01:34.406848 2664 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:34.406914 2664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:34.407097 2664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:34.409169 2664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "02899e54302b42eca417c15de6736877"
format_stamp: "Formatted at 2026-08-10 18:01:34 on dist-test-slave-srgv"
server_key: "1e73c4037f331ab26ebbfa6b5c1a98ba"
server_key_iv: "41d0da7d55f3c828e5c6902c4b7ca7b3"
server_key_version: "encryptionkey@0"
I20260810 18:01:34.409752 2664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "02899e54302b42eca417c15de6736877"
format_stamp: "Formatted at 2026-08-10 18:01:34 on dist-test-slave-srgv"
server_key: "1e73c4037f331ab26ebbfa6b5c1a98ba"
server_key_iv: "41d0da7d55f3c828e5c6902c4b7ca7b3"
server_key_version: "encryptionkey@0"
I20260810 18:01:34.413520 2664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:01:34.415855 2679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:34.417110 2664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:34.417249 2664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "02899e54302b42eca417c15de6736877"
format_stamp: "Formatted at 2026-08-10 18:01:34 on dist-test-slave-srgv"
server_key: "1e73c4037f331ab26ebbfa6b5c1a98ba"
server_key_iv: "41d0da7d55f3c828e5c6902c4b7ca7b3"
server_key_version: "encryptionkey@0"
I20260810 18:01:34.417362 2664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:34.429894 2664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:34.432761 2664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:34.432976 2664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:34.433657 2664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:34.434500 2664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:34.434547 2664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:34.434615 2664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:34.434652 2664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:34.443871 2664 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:42895
I20260810 18:01:34.443892 2797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:42895 every 8 connection(s)
I20260810 18:01:34.444849 2664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:34.448585 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 2664
I20260810 18:01:34.448693 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:34.448969 29005 external_mini_cluster.cc:1428] Setting key 3459ee29551930984491d0417630b290
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:34.459810 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:34.446788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:41741 (local address 127.28.83.126:45531)
0810 18:01:34.446971 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:34.446976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:34.447763 (+ 787us) server_negotiation.cc:407] Connection header received
0810 18:01:34.449306 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:34.449309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:34.449368 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:34.449504 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:34.450855 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.451466 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.453180 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.453400 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.453544 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:34.456226 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:34.456229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:34.456232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:34.456257 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:34.457613 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.458152 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.458159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.458161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.458268 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.458661 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.458666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.458668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.458861 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:34.458967 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:34.459509 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 18:01:34.459642 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 18:01:34.460672 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:34.447123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:45531 (local address 127.28.83.67:41741)
0810 18:01:34.447589 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:34.447636 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:34.449081 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:34.449689 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:34.449695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:34.450133 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:34.450704 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:34.450723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.451580 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.451583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:34.453011 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:34.453022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.453586 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.453590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:34.453736 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:34.454328 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:34.454344 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:34.456040 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:34.457733 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:34.457740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:34.457755 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:34.458039 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:34.458376 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:34.458379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:34.458381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:34.458560 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:34.458983 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:34.458992 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:34.459257 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 18:01:34.459451 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 18:01:34.459807 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260810 18:01:34.461968 2798 heartbeater.cc:347] Connected to a master server at 127.28.83.126:45531
I20260810 18:01:34.462236 2798 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:34.462816 2798 heartbeater.cc:511] Master 127.28.83.126:45531 requested a full tablet report, sending...
I20260810 18:01:34.463863 2327 ts_manager.cc:194] Registered new tserver with Master: 02899e54302b42eca417c15de6736877 (127.28.83.67:42895)
I20260810 18:01:34.464497 2327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:41741
I20260810 18:01:34.475232 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/test-admin.keytab.
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:34.491449 29005 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 18:01:34.494586 2805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:34.492883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.28.83.65:43097)
0810 18:01:34.493255 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:34.493272 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:34.493295 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:01:34.493343 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:34.493348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:34.493600 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:34.493732 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:34.494346 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52434: BlockingRecv error: recv got EOF from 127.0.0.1:52434 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":151,"threads_started":1}
W20260810 18:01:34.494589 29005 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:34.495772 29005 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:34.496690 29005 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:34.506232 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:34.497508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.28.83.65:43097)
0810 18:01:34.497652 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:34.497655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:34.497728 (+ 73us) server_negotiation.cc:407] Connection header received
0810 18:01:34.497811 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:34.497813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:34.497847 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:34.497912 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:34.498398 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.499147 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.499943 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.500294 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.500374 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:34.502561 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:34.502570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:34.502582 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:34.502604 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:34.504303 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.504667 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.504672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.504673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.504755 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.505053 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.505057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.505059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.505316 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:34.505550 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:34.505856 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 18:01:34.506016 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 18:01:34.509912 2805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:34.509007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.28.83.65:43097)
0810 18:01:34.509135 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:34.509138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:34.509161 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:01:34.509271 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:34.509273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:34.509306 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:34.509369 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:34.509799 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52450: BlockingRecv error: recv got EOF from 127.0.0.1:52450 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 18:01:34.509970 29005 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:34.510937 29005 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:34.511802 29005 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:34.519783 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:34.512542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.28.83.65:43097)
0810 18:01:34.512663 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:34.512666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:34.512680 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:01:34.512786 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:34.512788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:34.512820 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:34.512881 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:34.513311 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.513894 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.514615 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.514845 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.514919 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:34.516942 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:34.516945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:34.516947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:34.516968 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:34.518384 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.518749 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.518751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.518753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.518822 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.519046 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.519048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.519049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.519196 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:34.519290 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:34.519478 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 18:01:34.519593 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 18:01:34.522503 2805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:34.521574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.28.83.65:43097)
0810 18:01:34.521698 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:34.521702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:34.521758 (+ 56us) server_negotiation.cc:407] Connection header received
0810 18:01:34.521860 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:34.521862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:34.521912 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:34.522031 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:34.522397 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52464: BlockingRecv error: recv got EOF from 127.0.0.1:52464 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 18:01:34.522568 29005 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786384863806381-29005-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:34.523566 29005 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:34.524436 29005 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:01:34.532449 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:34.525212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.28.83.65:43097)
0810 18:01:34.525336 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:34.525339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:34.525355 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:01:34.525551 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:34.525554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:34.525589 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:34.525652 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:34.526077 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.526540 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.527273 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:34.527472 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:34.527543 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:34.529622 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:34.529625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:34.529627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:34.529647 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:34.531062 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.531423 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.531426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.531427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.531497 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:34.531732 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:34.531734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:34.531735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:34.531880 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:34.531972 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:34.532160 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 18:01:34.532271 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:01:34.533537 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 2392
I20260810 18:01:34.542412 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 2529
I20260810 18:01:34.548477 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 2664
I20260810 18:01:34.555094 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 2295
2026-08-10T18:01:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2241 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2818](info): set up 2 sockets
Aug 10 18:01:34 dist-test-slave-srgv krb5kdc[2818](info): commencing operation
krb5kdc: starting...
W20260810 18:01:35.613029 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:35 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.126.keytab.
Entry for principal oryx/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.126.keytab.
I20260810 18:01:35.738785 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:39527
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.126.keytab
--principal=oryx/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:39527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:35.850549 2831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:35.850894 2831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:35.850970 2831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:35.855165 2831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:35.855265 2831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:35.855304 2831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:35.855365 2831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:35.855403 2831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:35.860863 2831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:39527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.126.keytab
--principal=oryx/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:39527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:35.862104 2831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:35.863139 2831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:35.871927 2839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:35.871994 2837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:35.871927 2836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:35.871975 2831 server_base.cc:1034] running on GCE node
I20260810 18:01:35.872745 2831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:35.873904 2831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:35.875078 2831 hybrid_clock.cc:648] HybridClock initialized: now 1786384895875051 us; error 36 us; skew 500 ppm
Aug 10 18:01:35 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384895, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:35.877907 2831 init.cc:377] Logged in from keytab as oryx/127.28.83.126@KRBTEST.COM (short username oryx)
I20260810 18:01:35.878897 2831 webserver.cc:460] Webserver started at http://127.28.83.126:46295/ using document root <none> and password file <none>
I20260810 18:01:35.879472 2831 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:35.879567 2831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:35.879809 2831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:35.881980 2831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "e68f257ff67e4efebe5bb38e1fafe09b"
format_stamp: "Formatted at 2026-08-10 18:01:35 on dist-test-slave-srgv"
server_key: "30e617e93c2c7239e70e0a34ea6b3799"
server_key_iv: "795ca1003643943157798e3bc5c9bc82"
server_key_version: "encryptionkey@0"
I20260810 18:01:35.882571 2831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "e68f257ff67e4efebe5bb38e1fafe09b"
format_stamp: "Formatted at 2026-08-10 18:01:35 on dist-test-slave-srgv"
server_key: "30e617e93c2c7239e70e0a34ea6b3799"
server_key_iv: "795ca1003643943157798e3bc5c9bc82"
server_key_version: "encryptionkey@0"
I20260810 18:01:35.886516 2831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:01:35.888998 2847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:35.890255 2831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:35.890409 2831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "e68f257ff67e4efebe5bb38e1fafe09b"
format_stamp: "Formatted at 2026-08-10 18:01:35 on dist-test-slave-srgv"
server_key: "30e617e93c2c7239e70e0a34ea6b3799"
server_key_iv: "795ca1003643943157798e3bc5c9bc82"
server_key_version: "encryptionkey@0"
I20260810 18:01:35.890527 2831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:35.909868 2831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:35.912796 2831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:35.913002 2831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:35.919894 2831 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:39527
I20260810 18:01:35.920184 2901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:39527 every 8 connection(s)
I20260810 18:01:35.920864 2831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:35.923527 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:35.925940 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 2831
I20260810 18:01:35.926096 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:35.926406 29005 external_mini_cluster.cc:1428] Setting key 1acc3dc316065813cd24201ec0411db3
I20260810 18:01:35.929687 2902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: Bootstrap starting.
I20260810 18:01:35.931799 2902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:35.932519 2902 log.cc:824] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:35.934571 2902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: No bootstrap required, opened a new log
Aug 10 18:01:35 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:35.938141 2902 raft_consensus.cc:348] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } }
I20260810 18:01:35.938361 2902 raft_consensus.cc:374] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:35.938432 2902 raft_consensus.cc:729] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68f257ff67e4efebe5bb38e1fafe09b, State: Initialized, Role: FOLLOWER
I20260810 18:01:35.938913 2902 consensus_queue.cc:260] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } }
I20260810 18:01:35.939042 2902 raft_consensus.cc:388] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:35.939129 2902 raft_consensus.cc:482] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:35.939229 2902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:35.940146 2902 raft_consensus.cc:504] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } }
I20260810 18:01:35.940547 2902 leader_election.cc:302] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e68f257ff67e4efebe5bb38e1fafe09b; no voters:
I20260810 18:01:35.940941 2902 leader_election.cc:288] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:35.941020 2907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:35.941206 2907 raft_consensus.cc:686] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 1 LEADER]: Becoming Leader. State: Replica: e68f257ff67e4efebe5bb38e1fafe09b, State: Running, Role: LEADER
I20260810 18:01:35.941555 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:35.927823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.28.83.126:39527)
0810 18:01:35.928271 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:35.928287 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:35.928322 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:01:35.928942 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:35.928958 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:35.929273 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:35.929656 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:35.930467 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:35.931448 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:35.932552 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:35.933076 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:35.933251 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:35.936025 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:35.936030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:35.936042 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:35.936070 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:35.938375 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:35.938872 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:35.938883 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:35.938889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:35.939000 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:35.939367 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:35.939371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:35.939373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:35.939772 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:35.939986 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:35.940288 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 18:01:35.940581 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":156,"threads_started":1}
I20260810 18:01:35.941538 2907 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } }
I20260810 18:01:35.942256 2902 sys_catalog.cc:564] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:35.944301 2908 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68f257ff67e4efebe5bb38e1fafe09b. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } }
I20260810 18:01:35.944439 2908 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:35.945113 2921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:35.945160 2909 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } }
I20260810 18:01:35.945468 2909 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:35.948815 2921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:35.955612 2921 catalog_manager.cc:1347] Generated new cluster ID: bcfc9d0c64304b67b1ea56372f488b35
I20260810 18:01:35.955695 2921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:35.971632 2921 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:35.972937 2921 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:35.982944 2921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: Generated new TSK 0
I20260810 18:01:35.983804 2921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab.
Entry for principal oryx/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab.
I20260810 18:01:36.012765 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39527
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab
--principal=oryx/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:36.126452 2928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:36.126693 2928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:36.126734 2928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:36.130864 2928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:36.130915 2928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:36.131021 2928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:36.136132 2928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab
--principal=oryx/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:36.137226 2928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:36.138351 2928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:36.148432 2936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:36.148407 2933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:36.148396 2934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:36.148715 2928 server_base.cc:1034] running on GCE node
I20260810 18:01:36.149137 2928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:36.149857 2928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:36.151021 2928 hybrid_clock.cc:648] HybridClock initialized: now 1786384896151007 us; error 30 us; skew 500 ppm
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:36.153951 2928 init.cc:377] Logged in from keytab as oryx/127.28.83.65@KRBTEST.COM (short username oryx)
I20260810 18:01:36.154835 2928 webserver.cc:460] Webserver started at http://127.28.83.65:38145/ using document root <none> and password file <none>
I20260810 18:01:36.155397 2928 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:36.155493 2928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:36.155735 2928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:36.157915 2928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "63f939c8313248218f4d26b864ae4dce"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "6e2ea94953be2a4244ffa5b7f9005aac"
server_key_iv: "1c8ecf1a27a60974d8393eb325330e7f"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.158510 2928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "63f939c8313248218f4d26b864ae4dce"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "6e2ea94953be2a4244ffa5b7f9005aac"
server_key_iv: "1c8ecf1a27a60974d8393eb325330e7f"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.162462 2928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:01:36.165037 2944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.166327 2928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:36.166466 2928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "63f939c8313248218f4d26b864ae4dce"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "6e2ea94953be2a4244ffa5b7f9005aac"
server_key_iv: "1c8ecf1a27a60974d8393eb325330e7f"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.166584 2928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:36.182729 2928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:36.186394 2928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:36.186568 2928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:36.187134 2928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:36.187958 2928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:36.188009 2928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.188050 2928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:36.188071 2928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.196614 2928 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:42123
I20260810 18:01:36.196683 3060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:42123 every 8 connection(s)
I20260810 18:01:36.197659 2928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:36.199724 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 2928
I20260810 18:01:36.199872 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:36.200169 29005 external_mini_cluster.cc:1428] Setting key 44048363799400686ed58f9dd32a7086
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:36.212392 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.199438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:49987 (local address 127.28.83.126:39527)
0810 18:01:36.199612 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:36.199619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:36.200617 (+ 998us) server_negotiation.cc:407] Connection header received
0810 18:01:36.202423 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:36.202426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:36.202470 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:36.202544 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:36.204104 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.204658 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.205868 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.206106 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.206229 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.208675 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:36.208679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:36.208683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:36.208714 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:36.210297 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.210825 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.210828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.210830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.210916 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.211258 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.211263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.211265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.211453 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:36.211571 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:36.212068 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 18:01:36.212212 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:01:36.213183 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.199895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.65:49987)
0810 18:01:36.200442 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:36.200499 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:36.202186 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:36.202749 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:36.202760 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:36.203305 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:36.203929 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.203943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.204793 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.204796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.205742 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.205751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.206263 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.206267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.206437 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.207057 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:36.207073 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:36.208503 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:36.210422 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.210428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.210443 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.210710 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.211009 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.211013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.211015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.211155 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.211566 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:36.211572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:36.211830 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 18:01:36.212026 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 18:01:36.212351 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":124,"threads_started":1}
I20260810 18:01:36.214587 3061 heartbeater.cc:347] Connected to a master server at 127.28.83.126:39527
I20260810 18:01:36.214835 3061 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:36.215356 3061 heartbeater.cc:511] Master 127.28.83.126:39527 requested a full tablet report, sending...
I20260810 18:01:36.216854 2864 ts_manager.cc:194] Registered new tserver with Master: 63f939c8313248218f4d26b864ae4dce (127.28.83.65:42123)
I20260810 18:01:36.218053 2864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:49987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab.
Entry for principal oryx/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab.
I20260810 18:01:36.228838 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39527
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab
--principal=oryx/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:36.343119 3067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:36.343475 3067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:36.343521 3067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:36.349136 3067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:36.349215 3067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:36.349282 3067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:36.355237 3067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab
--principal=oryx/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:36.356321 3067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:36.357321 3067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:36.367096 3072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:36.367136 3075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:36.367142 3073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:36.367237 3067 server_base.cc:1034] running on GCE node
I20260810 18:01:36.367745 3067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:36.368393 3067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:36.369575 3067 hybrid_clock.cc:648] HybridClock initialized: now 1786384896369537 us; error 32 us; skew 500 ppm
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:36.372350 3067 init.cc:377] Logged in from keytab as oryx/127.28.83.66@KRBTEST.COM (short username oryx)
I20260810 18:01:36.373299 3067 webserver.cc:460] Webserver started at http://127.28.83.66:33787/ using document root <none> and password file <none>
I20260810 18:01:36.373922 3067 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:36.373989 3067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:36.374233 3067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:36.376339 3067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "6e032e99470087278216108151c3dcc5"
server_key_iv: "8a9d3e6abd1a974d5d38bdae7c9ce6f0"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.376925 3067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "6e032e99470087278216108151c3dcc5"
server_key_iv: "8a9d3e6abd1a974d5d38bdae7c9ce6f0"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.380739 3067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 18:01:36.383304 3082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.384713 3067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:36.384864 3067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "6e032e99470087278216108151c3dcc5"
server_key_iv: "8a9d3e6abd1a974d5d38bdae7c9ce6f0"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.384991 3067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:36.399577 3067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:36.402506 3067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:36.402722 3067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:36.403321 3067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:36.404175 3067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:36.404247 3067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.404318 3067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:36.404371 3067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.413813 3067 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:44715
I20260810 18:01:36.413837 3198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:44715 every 8 connection(s)
I20260810 18:01:36.414726 3067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:36.415856 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 3067
I20260810 18:01:36.415973 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:36.416291 29005 external_mini_cluster.cc:1428] Setting key 442904b36d2aad0da83c3aab7be9f6ef
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.67@KRBTEST.COM" created.
I20260810 18:01:36.430676 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.416713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:39661 (local address 127.28.83.126:39527)
0810 18:01:36.416869 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:36.416874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:36.417827 (+ 953us) server_negotiation.cc:407] Connection header received
0810 18:01:36.419279 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:36.419283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:36.419336 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:36.419423 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:36.420580 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.421113 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.422965 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.423294 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.423394 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.426470 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:36.426473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:36.426475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:36.426498 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:36.428066 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.428753 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.428761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.428762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.428837 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.429201 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.429207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.429212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.429541 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:36.429684 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:36.430347 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 18:01:36.430486 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:01:36.431504 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.417139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.66:39661)
0810 18:01:36.417670 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:36.417730 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:36.419047 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:36.419602 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:36.419608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:36.419966 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:36.420434 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.420446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.421247 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.421253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.422798 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.422813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.423480 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.423486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.423712 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.424412 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:36.424437 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:36.426289 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:36.428209 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.428215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.428231 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.428636 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.428933 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.428937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.428939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.429085 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.429681 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:36.429689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:36.430049 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 18:01:36.430285 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 18:01:36.430627 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":119,"threads_started":1}
I20260810 18:01:36.432755 3199 heartbeater.cc:347] Connected to a master server at 127.28.83.126:39527
I20260810 18:01:36.433034 3199 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:36.433683 3199 heartbeater.cc:511] Master 127.28.83.126:39527 requested a full tablet report, sending...
I20260810 18:01:36.434854 2864 ts_manager.cc:194] Registered new tserver with Master: 7fa7b864070f42dbaf8af4a9ba72a0a8 (127.28.83.66:44715)
I20260810 18:01:36.435451 2864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:39661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab.
Entry for principal oryx/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab.
I20260810 18:01:36.446697 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39527
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab
--principal=oryx/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:36.562856 3204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:36.563149 3204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:36.563194 3204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:36.567344 3204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:36.567420 3204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:36.567538 3204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:36.573400 3204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab
--principal=oryx/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:39527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:36.574590 3204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:36.575681 3204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:36.585868 3209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:36.585870 3212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:36.585915 3204 server_base.cc:1034] running on GCE node
W20260810 18:01:36.586124 3210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:36.586557 3204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:36.587162 3204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:36.588341 3204 hybrid_clock.cc:648] HybridClock initialized: now 1786384896588312 us; error 28 us; skew 500 ppm
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:36.591388 3204 init.cc:377] Logged in from keytab as oryx/127.28.83.67@KRBTEST.COM (short username oryx)
I20260810 18:01:36.592309 3204 webserver.cc:460] Webserver started at http://127.28.83.67:35893/ using document root <none> and password file <none>
I20260810 18:01:36.592865 3204 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:36.592928 3204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:36.593195 3204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:36.595671 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "b48a1c3a95094db1a8be66645ecfccb7"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "48179fb9aec99d017fe4d6ebdb9a4eb3"
server_key_iv: "560c426dc2f37cc195f287137186a914"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.596271 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "b48a1c3a95094db1a8be66645ecfccb7"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "48179fb9aec99d017fe4d6ebdb9a4eb3"
server_key_iv: "560c426dc2f37cc195f287137186a914"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.599990 3204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:01:36.602804 3219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.604106 3204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:36.604207 3204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "b48a1c3a95094db1a8be66645ecfccb7"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "48179fb9aec99d017fe4d6ebdb9a4eb3"
server_key_iv: "560c426dc2f37cc195f287137186a914"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.604353 3204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:36.636745 3204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:36.640027 3204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:36.640277 3204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:36.640928 3204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:36.641909 3204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:36.641961 3204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.642025 3204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:36.642057 3204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.651521 3204 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:35521
I20260810 18:01:36.651546 3337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:35521 every 8 connection(s)
I20260810 18:01:36.652467 3204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:36.654279 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 3204
I20260810 18:01:36.654403 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:36.654702 29005 external_mini_cluster.cc:1428] Setting key 623db59384e3b72b55cefcc1f1b06499
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:36.667198 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.654450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:52467 (local address 127.28.83.126:39527)
0810 18:01:36.654605 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:36.654610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:36.655714 (+ 1104us) server_negotiation.cc:407] Connection header received
0810 18:01:36.657554 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:36.657557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:36.657620 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:36.657716 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:36.658897 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.659443 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.660589 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.660820 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.660933 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.663560 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:36.663564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:36.663567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:36.663609 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:36.664994 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.665623 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.665627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.665629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.665719 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.666087 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.666090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.666092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.666281 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:36.666394 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:36.666896 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 18:01:36.667028 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:01:36.668238 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.654957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.67:52467)
0810 18:01:36.655539 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:36.655600 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:36.657314 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:36.657903 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:36.657910 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:36.658278 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:36.658750 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.658764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.659574 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.659577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.660456 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.660465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.660976 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.660979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.661152 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.661838 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:36.661854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:36.663391 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:36.665122 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.665128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.665144 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.665499 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.665844 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.665846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.665847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.665980 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.666392 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:36.666399 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:36.666650 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 18:01:36.666837 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 18:01:36.667180 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":170,"threads_started":1}
I20260810 18:01:36.669574 3338 heartbeater.cc:347] Connected to a master server at 127.28.83.126:39527
I20260810 18:01:36.669785 3338 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:36.670323 3338 heartbeater.cc:511] Master 127.28.83.126:39527 requested a full tablet report, sending...
I20260810 18:01:36.671414 2864 ts_manager.cc:194] Registered new tserver with Master: b48a1c3a95094db1a8be66645ecfccb7 (127.28.83.67:35521)
I20260810 18:01:36.671988 2864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:52467
I20260810 18:01:36.680861 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.83.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.125.keytab.
Entry for principal oryx/127.28.83.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.125.keytab.
I20260810 18:01:36.708748 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.125:42411
--webserver_interface=127.28.83.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.125.keytab
--principal=oryx/127.28.83.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:39527,127.28.83.125:42411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:36.824682 3343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:36.825004 3343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:36.825088 3343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:36.829092 3343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:36.829191 3343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:36.829226 3343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:36.829276 3343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:36.829308 3343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:36.834628 3343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:39527,127.28.83.125:42411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.125.keytab
--principal=oryx/127.28.83.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.125:42411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.83.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
W20260810 18:01:36.834903 3343 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.83.126:39527,127.28.83.125:42411'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 18:01:36.835841 3343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:36.836911 3343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:36.845748 3349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:36.845917 3343 server_base.cc:1034] running on GCE node
W20260810 18:01:36.845752 3348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:36.845752 3351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:36.846753 3343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:36.847862 3343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:36.849023 3343 hybrid_clock.cc:648] HybridClock initialized: now 1786384896848998 us; error 41 us; skew 500 ppm
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:36.852038 3343 init.cc:377] Logged in from keytab as oryx/127.28.83.125@KRBTEST.COM (short username oryx)
I20260810 18:01:36.853029 3343 webserver.cc:460] Webserver started at http://127.28.83.125:35423/ using document root <none> and password file <none>
I20260810 18:01:36.853713 3343 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:36.853807 3343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:36.854045 3343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:36.856153 3343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data/instance:
uuid: "55c799d9947b47f0a67d47b3053da1ff"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "0c5fe27b5f4f445c9fe445985f24aab8"
server_key_iv: "964b1c99c1bb3e2a859fd654380d4611"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.856776 3343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal/instance:
uuid: "55c799d9947b47f0a67d47b3053da1ff"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "0c5fe27b5f4f445c9fe445985f24aab8"
server_key_iv: "964b1c99c1bb3e2a859fd654380d4611"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.860677 3343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:01:36.863199 3358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:36.864478 3343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:36.864637 3343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal
uuid: "55c799d9947b47f0a67d47b3053da1ff"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "0c5fe27b5f4f445c9fe445985f24aab8"
server_key_iv: "964b1c99c1bb3e2a859fd654380d4611"
server_key_version: "encryptionkey@0"
I20260810 18:01:36.864768 3343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:36.879127 3343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:36.882598 3343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:36.882848 3343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:36.890434 3343 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.125:42411
I20260810 18:01:36.890733 3414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.125:42411 every 8 connection(s)
I20260810 18:01:36.891503 3343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
I20260810 18:01:36.894435 3415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:36.895853 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 3343
I20260810 18:01:36.898794 3415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.83.125@KRBTEST.COM
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.125@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:36.910920 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.900740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.28.83.126:39527)
0810 18:01:36.900885 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:36.900891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:36.901207 (+ 316us) server_negotiation.cc:407] Connection header received
0810 18:01:36.901388 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:36.901397 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:36.901494 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:36.901577 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:36.902427 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.903094 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.904219 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.904457 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.904538 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.907132 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:36.907138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:36.907143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:36.907174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:36.908791 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.909319 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.909324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.909329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.909438 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.909796 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.909802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.909805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.910086 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:36.910195 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:36.910594 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 18:01:36.910743 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:01:36.911746 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.897515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.28.83.125:42411)
0810 18:01:36.897936 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:36.897950 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:36.897984 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:01:36.898605 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:36.898620 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:36.898926 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:36.899269 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:36.900151 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.901632 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.902737 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.903485 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.903597 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.906575 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:36.906583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:36.906603 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:36.906625 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:36.908690 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.909246 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.909271 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.909276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.909438 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.909736 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.909738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.909739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.910296 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:36.910451 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:36.910730 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 18:01:36.911129 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":198,"server-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
I20260810 18:01:36.911744 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.900674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:49974)
0810 18:01:36.901077 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:36.901121 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:36.901265 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:36.901635 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:36.901640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:36.902143 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:36.902296 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.902304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.903226 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.903229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.904070 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.904079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.904618 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.904625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.904769 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.905300 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:36.905325 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:36.906975 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:36.908905 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.908915 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.908926 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.909213 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.909538 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.909541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.909543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.909686 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.910195 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:36.910200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:36.910324 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 18:01:36.910525 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 18:01:36.910881 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":156,"threads_started":1}
I20260810 18:01:36.913208 3415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.125@KRBTEST.COM for oryx/127.28.83.125@KRBTEST.COM
I20260810 18:01:36.921240 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.913745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.0.0.1:56452)
0810 18:01:36.913833 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:36.913864 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:36.913943 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:36.914172 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:36.914174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:36.914326 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:36.914403 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.914408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.915475 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.915481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.916078 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.916105 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.916579 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.916582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.916677 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.917063 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:36.917073 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:36.918343 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:36.919804 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.919808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.919810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.919993 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.920272 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.920275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.920277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.920352 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.920710 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:36.920713 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:36.920799 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 18:01:36.920904 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 18:01:36.921037 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 18:01:36.921240 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.913814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.28.83.125:42411)
0810 18:01:36.913922 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:36.913925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:36.913937 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:01:36.914030 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:36.914032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:36.914066 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:36.914134 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:36.914509 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.915363 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.916236 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.916475 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.916590 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.918474 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:36.918478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:36.918480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:36.918503 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:36.919697 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.920097 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.920100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.920102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.920178 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.920436 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.920442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.920444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.920616 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:36.920721 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:36.920949 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 18:01:36.921093 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":46}
I20260810 18:01:36.924428 3415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: Bootstrap starting.
I20260810 18:01:36.926755 3415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:36.927479 3415 log.cc:824] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:36.929502 3415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: No bootstrap required, opened a new log
I20260810 18:01:36.932487 3415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } }
I20260810 18:01:36.932684 3415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:36.932754 3415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c799d9947b47f0a67d47b3053da1ff, State: Initialized, Role: FOLLOWER
I20260810 18:01:36.933207 3415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } }
I20260810 18:01:36.933741 3422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } } }
I20260810 18:01:36.933887 3422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:01:36.934423 3415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:01:36.939090 3433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:01:36.939195 3433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:01:36.942065 3343 master_runner.cc:428] Detected that this master 55c799d9947b47f0a67d47b3053da1ff is joining an existing cluster
I20260810 18:01:36.942139 3343 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.83.125:42411
I20260810 18:01:36.957600 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.944976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.28.83.125:42411)
0810 18:01:36.945148 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:36.945179 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:36.945820 (+ 641us) server_negotiation.cc:407] Connection header received
0810 18:01:36.945920 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:36.945923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:36.945995 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:36.946093 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:36.949047 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.949907 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.951243 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.951698 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.951799 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.953975 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:36.953978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:36.953981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:36.954004 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:36.955829 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.956286 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.956289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.956291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.956358 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.956596 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.956599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.956600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.956872 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:36.956943 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:36.957229 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 18:01:36.957351 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 18:01:36.958128 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.944565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.0.0.1:56462)
0810 18:01:36.945664 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:36.945699 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:36.945811 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:36.948496 (+ 2685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:36.948499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:36.948756 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:36.948918 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.948926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.950066 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.950071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.951123 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.951133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.951816 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.951819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.951915 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.953144 (+ 1229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:36.953164 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:36.953838 (+ 674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:36.955951 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.955954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.955956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.956192 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.956446 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.956448 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.956450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.956518 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.957000 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:36.957003 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:36.957079 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 18:01:36.957817 (+ 738us) client_negotiation.cc:240] Negotiation successful
0810 18:01:36.957964 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":138,"threads_started":1}
I20260810 18:01:36.971273 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.945199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.28.83.126:39527)
0810 18:01:36.945331 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:36.945338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:36.957830 (+ 12492us) server_negotiation.cc:407] Connection header received
0810 18:01:36.957904 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:36.957908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:36.957985 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:36.958066 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:36.958752 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.959617 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.962827 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.963207 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.963341 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.964894 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:36.964898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:36.964901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:36.964930 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:36.969003 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.969729 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.969734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.969736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.969841 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.970218 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.970221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.970223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.970462 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:36.970545 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:36.970913 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 18:01:36.971052 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:01:36.971875 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.945439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:49988)
0810 18:01:36.957679 (+ 12240us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:36.957697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:36.957802 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:36.958193 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:36.958197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:36.958447 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:36.958614 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.958624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.961493 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.961497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.962659 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.962671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.963368 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.963372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.963492 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.964234 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:36.964249 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:36.964725 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:36.969149 (+ 4424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.969152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.969154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.969589 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.969973 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.969976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.969979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.970063 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.970602 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:36.970605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:36.970722 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 18:01:36.971498 (+ 776us) client_negotiation.cc:240] Negotiation successful
0810 18:01:36.971694 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12155,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.28.83.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.124@KRBTEST.COM" created.
Aug 10 18:01:36 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384895, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.126@KRBTEST.COM for oryx/127.28.83.125@KRBTEST.COM
I20260810 18:01:36.988339 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.977812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.28.83.125:42411)
0810 18:01:36.977917 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:36.977922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:36.977935 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:01:36.978002 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:36.978004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:36.978056 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:36.978134 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:36.978975 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.979844 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.980992 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.981326 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.981465 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.984247 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:36.984252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:36.984254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:36.984283 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:36.986147 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.986784 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.986787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.986789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.986877 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:36.987225 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:36.987228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:36.987230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:36.987452 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:36.987554 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:36.988001 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 18:01:36.988154 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:01:36.988415 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:36.977026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.0.0.1:56468)
0810 18:01:36.977537 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:36.977692 (+ 155us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:36.977839 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:36.978201 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:36.978208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:36.978704 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:36.978849 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.978856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.979962 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.979964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.980856 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:36.980870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:36.981475 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:36.981481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:36.981616 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:36.982390 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:36.982403 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:36.984088 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:36.986279 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.986293 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.986303 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.986665 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.986989 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:36.986994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:36.986999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:36.987115 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:36.987579 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:36.987587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:36.987753 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 18:01:36.987927 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 18:01:36.988163 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
I20260810 18:01:36.990201 2864 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.83.125:42411: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff"
member_type: NON_VOTER
last_known_addr {
host: "127.28.83.125"
port: 42411
}
attrs {
promote: true
}
}
dest_uuid: "e68f257ff67e4efebe5bb38e1fafe09b"
cas_config_opid_index: -1
I20260810 18:01:36.991019 2864 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } }
I20260810 18:01:36.993117 3445 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 55c799d9947b47f0a67d47b3053da1ff (127.28.83.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } } }
I20260810 18:01:36.993991 3386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:36.995071 3445 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.83.125:42411
I20260810 18:01:36.995323 3445 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } } }
I20260810 18:01:36.995438 3445 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:36.995437 3386 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Refusing update from remote peer e68f257ff67e4efebe5bb38e1fafe09b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 18:01:36.996197 3343 master.cc:537] Master@127.28.83.125:42411 shutting down...
I20260810 18:01:36.997890 3446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:01:37.000385 2850 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.124.keytab.
Entry for principal oryx/127.28.83.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.124.keytab.
I20260810 18:01:37.001544 3343 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:37.001878 3343 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:37.001958 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.124:37029
--webserver_interface=127.28.83.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.124.keytab
--principal=oryx/127.28.83.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:39527,127.28.83.125:42411,127.28.83.124:37029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:37.002079 2850 consensus_peers.cc:487] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b -> Peer 55c799d9947b47f0a67d47b3053da1ff (127.28.83.125:42411): Couldn't send request to peer 55c799d9947b47f0a67d47b3053da1ff. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:01:37.002315 3343 tablet_replica.cc:330] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: stopping tablet replica
I20260810 18:01:37.017299 3343 master.cc:552] Master@127.28.83.125:42411 shutdown complete.
I20260810 18:01:37.017383 3343 master_runner.cc:305] Clearing existing system tablet
I20260810 18:01:37.017666 3343 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:37.017745 3343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:37.018882 3343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:37.019733 3451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.019979 3343 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:37.020121 3343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:37.020157 3343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal
uuid: "55c799d9947b47f0a67d47b3053da1ff"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "0c5fe27b5f4f445c9fe445985f24aab8"
server_key_iv: "964b1c99c1bb3e2a859fd654380d4611"
server_key_version: "encryptionkey@0"
I20260810 18:01:37.021945 3343 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:37.023783 3343 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 18:01:37.023895 3343 log.cc:1196] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 18:01:37.024145 3343 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: Deleting consensus metadata
I20260810 18:01:37.024336 3343 master_runner.cc:315] Copying system tablet from 127.28.83.126:39527
I20260810 18:01:37.025677 3343 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: tablet copy: Beginning tablet copy session from remote peer at address 127.28.83.126:39527
I20260810 18:01:37.037202 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.026405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.28.83.126:39527)
0810 18:01:37.026559 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.026564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.029721 (+ 3157us) server_negotiation.cc:407] Connection header received
0810 18:01:37.029881 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.029884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.029938 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.030024 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.030469 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.031007 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.031825 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.032058 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.032169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.033390 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.033394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.033397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.033458 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.035171 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.035740 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.035744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.035745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.035834 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.036208 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.036211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.036213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.036481 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.036588 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.036861 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.037070 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:01:37.037184 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.026313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50004)
0810 18:01:37.029599 (+ 3286us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.029614 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.029758 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.030041 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.030044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.030259 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.030347 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.030353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.031120 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.031125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.031704 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.031713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.032178 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.032182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.032299 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.032872 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.032884 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.033244 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.035288 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.035292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.035294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.035576 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.035933 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.035936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.035937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.036117 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.036577 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.036581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.036690 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.036804 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.036965 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3214,"thread_start_us":87,"threads_started":1}
I20260810 18:01:37.038199 2886 tablet_copy_service.cc:140] P e68f257ff67e4efebe5bb38e1fafe09b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 55c799d9947b47f0a67d47b3053da1ff ({username='oryx', principal='oryx/127.28.83.125@KRBTEST.COM'} at 127.0.0.1:50004)
I20260810 18:01:37.038385 2886 tablet_copy_service.cc:161] P e68f257ff67e4efebe5bb38e1fafe09b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 55c799d9947b47f0a67d47b3053da1ff at {username='oryx', principal='oryx/127.28.83.125@KRBTEST.COM'} at 127.0.0.1:50004: session id = 55c799d9947b47f0a67d47b3053da1ff-00000000000000000000000000000000
I20260810 18:01:37.039871 2886 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:01:37.041626 3343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 18:01:37.044014 3343 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: tablet copy: Starting download of 0 data blocks...
I20260810 18:01:37.044222 3343 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: tablet copy: Starting download of 1 WAL segments...
I20260810 18:01:37.046103 3343 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:01:37.047791 2886 tablet_copy_service.cc:342] P e68f257ff67e4efebe5bb38e1fafe09b: Request end of tablet copy session 55c799d9947b47f0a67d47b3053da1ff-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.83.125@KRBTEST.COM'} at 127.0.0.1:50004
I20260810 18:01:37.047928 2886 tablet_copy_service.cc:433] P e68f257ff67e4efebe5bb38e1fafe09b: ending tablet copy session 55c799d9947b47f0a67d47b3053da1ff-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 55c799d9947b47f0a67d47b3053da1ff
I20260810 18:01:37.055742 3343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:37.057446 3461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:37.057687 3459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:37.057924 3343 server_base.cc:1034] running on GCE node
W20260810 18:01:37.061858 3458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:37.062098 3343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:37.062458 3343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:37.063602 3343 hybrid_clock.cc:648] HybridClock initialized: now 1786384897063589 us; error 37 us; skew 500 ppm
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:37.065196 3343 init.cc:377] Logged in from keytab as oryx/127.28.83.125@KRBTEST.COM (short username oryx)
I20260810 18:01:37.065743 3343 webserver.cc:460] Webserver started at http://127.28.83.125:43857/ using document root <none> and password file <none>
I20260810 18:01:37.065973 3343 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:37.066030 3343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:37.067368 3343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:37.068533 3468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.068778 3343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:37.068858 3343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal
uuid: "55c799d9947b47f0a67d47b3053da1ff"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "0c5fe27b5f4f445c9fe445985f24aab8"
server_key_iv: "964b1c99c1bb3e2a859fd654380d4611"
server_key_version: "encryptionkey@0"
I20260810 18:01:37.068945 3343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:37.077900 3343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:37.080091 3343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:37.085230 3343 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.125:42411
I20260810 18:01:37.085367 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.125:42411 every 8 connection(s)
I20260810 18:01:37.085871 3343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
I20260810 18:01:37.087473 3523 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:01:37.088518 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: Bootstrap starting.
I20260810 18:01:37.097365 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:01:37.097743 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: Bootstrap complete.
I20260810 18:01:37.098341 3523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } }
I20260810 18:01:37.098483 3523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55c799d9947b47f0a67d47b3053da1ff, State: Initialized, Role: LEARNER
I20260810 18:01:37.098595 3523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } }
I20260810 18:01:37.098946 3526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } } }
I20260810 18:01:37.099108 3523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:37.099112 3526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: This master's current role is: LEARNER
I20260810 18:01:37.102510 3537 catalog_manager.cc:1259] Loaded cluster ID: bcfc9d0c64304b67b1ea56372f488b35
I20260810 18:01:37.102584 3537 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: loading cluster ID for follower catalog manager: success
I20260810 18:01:37.104076 3537 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: acquiring CA information for follower catalog manager: success
I20260810 18:01:37.105144 3537 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 18:01:37.161567 3449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:37.161851 3449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:37.161893 3449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:37.165947 3449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:37.166004 3449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:37.166024 3449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:37.166040 3449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:37.166056 3449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:37.171805 3449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:39527,127.28.83.125:42411,127.28.83.124:37029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.124.keytab
--principal=oryx/127.28.83.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.124:37029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.83.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:37.172854 3449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:37.173849 3449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:37.182726 3545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:37.182809 3449 server_base.cc:1034] running on GCE node
W20260810 18:01:37.182739 3542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:37.182739 3543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:37.183583 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:37.184818 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:37.185995 3449 hybrid_clock.cc:648] HybridClock initialized: now 1786384897185936 us; error 86 us; skew 500 ppm
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:37.189188 3449 init.cc:377] Logged in from keytab as oryx/127.28.83.124@KRBTEST.COM (short username oryx)
I20260810 18:01:37.190328 3449 webserver.cc:460] Webserver started at http://127.28.83.124:37135/ using document root <none> and password file <none>
I20260810 18:01:37.190938 3449 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:37.191052 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:37.191299 3449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:37.193676 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data/instance:
uuid: "2c754725a56f48c59f4fa09998b76521"
format_stamp: "Formatted at 2026-08-10 18:01:37 on dist-test-slave-srgv"
server_key: "91c99584a77fcdab0d8d263bdfdfc96a"
server_key_iv: "fd303c67fcaa8c58a2374a1369a4bfe0"
server_key_version: "encryptionkey@0"
I20260810 18:01:37.194289 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal/instance:
uuid: "2c754725a56f48c59f4fa09998b76521"
format_stamp: "Formatted at 2026-08-10 18:01:37 on dist-test-slave-srgv"
server_key: "91c99584a77fcdab0d8d263bdfdfc96a"
server_key_iv: "fd303c67fcaa8c58a2374a1369a4bfe0"
server_key_version: "encryptionkey@0"
I20260810 18:01:37.198357 3449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:01:37.201262 3552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.202570 3449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:37.202726 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal
uuid: "2c754725a56f48c59f4fa09998b76521"
format_stamp: "Formatted at 2026-08-10 18:01:37 on dist-test-slave-srgv"
server_key: "91c99584a77fcdab0d8d263bdfdfc96a"
server_key_iv: "fd303c67fcaa8c58a2374a1369a4bfe0"
server_key_version: "encryptionkey@0"
I20260810 18:01:37.202844 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:37.220299 3061 heartbeater.cc:503] Master 127.28.83.126:39527 was elected leader, sending a full tablet report...
I20260810 18:01:37.234349 3449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:37.237399 3449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:37.237661 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:37.246402 3449 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.124:37029
I20260810 18:01:37.246722 3607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.124:37029 every 8 connection(s)
I20260810 18:01:37.247469 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
I20260810 18:01:37.250309 3608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:37.250340 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 3449
I20260810 18:01:37.255668 3608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.83.124@KRBTEST.COM
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.124@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:37.266655 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.251849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.28.83.124:37029)
0810 18:01:37.252256 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.252272 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.252306 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:01:37.252983 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.252999 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.253327 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.253701 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.254647 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.255651 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.256817 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.257602 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.257779 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.261083 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.261088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.261097 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.261125 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.263065 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.263565 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.263570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.263574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.263725 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.264015 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.264020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.264021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.264578 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.264730 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.265143 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.265496 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260810 18:01:37.268409 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.257816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.28.83.126:39527)
0810 18:01:37.258075 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.258080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.258670 (+ 590us) server_negotiation.cc:407] Connection header received
0810 18:01:37.258825 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.258827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.258875 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.258938 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.259994 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.260794 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.261933 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.262297 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.262422 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.264940 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.264946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.264949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.264984 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.266530 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.267092 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.267096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.267097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.267199 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.267455 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.267458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.267460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.267694 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.267823 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.268197 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.268300 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260810 18:01:37.268550 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.257665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50012)
0810 18:01:37.258508 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.258567 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.258711 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.259031 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.259041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.259565 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.259849 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.259857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.260943 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.260949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.261811 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.261821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.262424 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.262427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.262512 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.262996 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.263008 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.264776 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.266648 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.266655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.266660 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.266994 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.267294 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.267296 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.267297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.267369 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.267840 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.267846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.267971 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.268129 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.268309 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":103,"threads_started":1}
I20260810 18:01:37.269641 3608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.124@KRBTEST.COM for oryx/127.28.83.125@KRBTEST.COM
I20260810 18:01:37.277963 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.270260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.0.0.1:56472)
0810 18:01:37.270413 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.270427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.270500 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.270980 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.270983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.271125 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.271219 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.271224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.272087 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.272089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.272615 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.272622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.273026 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.273028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.273101 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.273606 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.273617 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.274802 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.276463 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.276466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.276467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.276661 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.276956 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.276959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.276960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.277046 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.277548 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.277551 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.277633 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.277730 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.277836 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 18:01:37.278008 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.270384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.28.83.125:42411)
0810 18:01:37.270669 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.270673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.270685 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:01:37.270724 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.270726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.270764 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.270921 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.271324 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.271985 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.272717 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.272933 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.273007 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.274940 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.274943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.274945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.274969 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.276358 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.276747 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.276750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.276752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.276823 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.277165 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.277167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.277168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.277390 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.277518 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.277784 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.277888 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
I20260810 18:01:37.278962 3608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.124@KRBTEST.COM for oryx/127.28.83.124@KRBTEST.COM
I20260810 18:01:37.286763 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.279422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.0.0.1:33692)
0810 18:01:37.279532 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.279547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.279633 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.279850 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.279853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.279983 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.280059 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.280065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.281141 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.281143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.281728 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.281737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.282143 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.282145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.282201 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.282616 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.282627 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.283797 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.285359 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.285362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.285363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.285710 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.285960 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.285962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.285964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.286037 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.286335 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.286337 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.286408 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.286485 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.286586 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 18:01:37.286777 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.279488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.28.83.124:37029)
0810 18:01:37.279602 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.279608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.279626 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:01:37.279690 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.279692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.279730 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.279794 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.280158 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.280676 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.281827 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.282048 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.282125 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.283916 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.283919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.283921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.283941 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.285263 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.285799 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.285802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.285803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.285876 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.286094 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.286096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.286098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.286253 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.286330 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.286513 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.286645 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":47}
I20260810 18:01:37.289875 3608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Bootstrap starting.
I20260810 18:01:37.292196 3608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:37.292907 3608 log.cc:824] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:37.295104 3608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: No bootstrap required, opened a new log
I20260810 18:01:37.298382 3608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } }
I20260810 18:01:37.298625 3608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:37.298664 3608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c754725a56f48c59f4fa09998b76521, State: Initialized, Role: FOLLOWER
I20260810 18:01:37.299098 3608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } }
I20260810 18:01:37.299640 3614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } } }
I20260810 18:01:37.299794 3614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:01:37.300428 3608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:01:37.305835 3625 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:01:37.305927 3625 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:01:37.309450 3449 master_runner.cc:428] Detected that this master 2c754725a56f48c59f4fa09998b76521 is joining an existing cluster
I20260810 18:01:37.309531 3449 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.83.124:37029
I20260810 18:01:37.320401 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 2928
I20260810 18:01:37.320721 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.311657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.28.83.126:39527)
0810 18:01:37.311797 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.311804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.312088 (+ 284us) server_negotiation.cc:407] Connection header received
0810 18:01:37.312302 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.312309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.312366 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.312426 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.313352 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.314179 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.315454 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.315789 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.315861 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.317271 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.317275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.317278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.317307 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.318908 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.319324 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.319328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.319330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.319423 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.319746 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.319749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.319751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.319950 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.320063 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.320454 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.320568 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:01:37.320731 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.311677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50024)
0810 18:01:37.311994 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.312012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.312160 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.312589 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.312592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.312894 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.313193 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.313213 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.314317 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.314320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.315320 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.315331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.316038 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.316041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.316190 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.316777 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.316794 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.317126 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.319029 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.319031 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.319033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.319230 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.319549 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.319552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.319554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.319646 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.320052 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.320054 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.320271 (+ 217us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.320397 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.320535 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":44,"spinlock_wait_cycles":5376,"thread_start_us":123,"threads_started":1}
I20260810 18:01:37.325634 3614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:37.325810 3614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } }
I20260810 18:01:37.326707 3614 leader_election.cc:288] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e68f257ff67e4efebe5bb38e1fafe09b (127.28.83.126:39527), 55c799d9947b47f0a67d47b3053da1ff (127.28.83.125:42411)
I20260810 18:01:37.327311 2875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c754725a56f48c59f4fa09998b76521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" is_pre_election: true
I20260810 18:01:37.333626 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c754725a56f48c59f4fa09998b76521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55c799d9947b47f0a67d47b3053da1ff" is_pre_election: true
I20260810 18:01:37.333830 3495 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 LEARNER]: Handling vote request from an unknown peer 2c754725a56f48c59f4fa09998b76521
I20260810 18:01:37.333961 3495 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 2c754725a56f48c59f4fa09998b76521 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260810 18:01:37.335407 3555 leader_election.cc:302] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c754725a56f48c59f4fa09998b76521; no voters: 55c799d9947b47f0a67d47b3053da1ff, e68f257ff67e4efebe5bb38e1fafe09b
I20260810 18:01:37.335932 3614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:37.339329 3614 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:01:37.340055 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:42123
--local_ip_for_outbound_sockets=127.28.83.65
--tserver_master_addrs=127.28.83.126:39527
--webserver_port=38145
--webserver_interface=127.28.83.65
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab
--principal=oryx/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.83.126:39527,127.28.83.125:42411
--tserver_master_addrs=127.28.83.126:39527,127.28.83.125:42411,127.28.83.124:37029 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384895, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.126@KRBTEST.COM for oryx/127.28.83.124@KRBTEST.COM
I20260810 18:01:37.346451 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.311714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.0.0.1:56476)
0810 18:01:37.312167 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.312178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.312265 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.313182 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.313185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.313512 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.313685 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.313693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.314806 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.314811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.315738 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.315748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.316484 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.316488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.316613 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.341738 (+ 25125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.341754 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.342179 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.344327 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.344330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.344332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.344698 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.345053 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.345056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.345058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.345181 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.345764 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.345767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.345879 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.345982 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.346139 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":1,"spinlock_wait_cycles":8320,"thread_start_us":229,"threads_started":1}
I20260810 18:01:37.347047 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.312600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.28.83.125:42411)
0810 18:01:37.312848 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.312853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.312868 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:01:37.312910 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.312915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.312968 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.313052 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.313837 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.314657 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.315884 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.316243 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.316326 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.342377 (+ 26051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.342381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.342384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.342416 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.344192 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.344836 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.344839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.344841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.344940 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.345305 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.345310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.345312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.345598 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.345715 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.346708 (+ 993us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.346862 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260810 18:01:37.348176 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.324311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.0.0.1:33720)
0810 18:01:37.324508 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.324524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.324598 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.336208 (+ 11610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.336213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.336560 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.336774 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.336786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.338338 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.338341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.339300 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.339315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.340063 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.340070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.340194 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.340756 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.340781 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.342652 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.345893 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.345897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.345899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.346162 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.346773 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.346777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.346778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.346875 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.347573 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.347577 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.347704 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.347803 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.347987 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 18:01:37.349103 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.324342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.28.83.124:37029)
0810 18:01:37.335447 (+ 11105us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.335458 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.335478 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:01:37.335708 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.335712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.335830 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.336939 (+ 1109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.337153 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.338127 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.339538 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.339917 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.342810 (+ 2893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.343002 (+ 192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.343009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.343013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.343039 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.345774 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.346293 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.346296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.346298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.346400 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.346992 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.346995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.346996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.347454 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.348551 (+ 1097us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.348624 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.348751 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10965,"thread_start_us":171,"threads_started":1}
I20260810 18:01:37.349491 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.312712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.28.83.124:37029)
0810 18:01:37.312816 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.312821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.312832 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:01:37.312859 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.312863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.312904 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.312975 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.313821 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.314507 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.315536 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.315848 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.315908 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.341633 (+ 25725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.341639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.341641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.341670 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.344946 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.348268 (+ 3322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.348271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.348273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.348360 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.348623 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.348626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.348628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.348816 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.348893 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.349197 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.349312 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 18:01:37.349987 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.312213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.0.0.1:33704)
0810 18:01:37.312517 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.312531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.312643 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.313084 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.313086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.313502 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.313707 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.313715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.314619 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.314622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.315440 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.315448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.316038 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.316041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.316194 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.340921 (+ 24727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.340939 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.341482 (+ 543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.345057 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.345059 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.345061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.348160 (+ 3099us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.348463 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.348466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.348468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.348539 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.348936 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.348938 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.349023 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.349650 (+ 627us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.349802 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":21,"spinlock_wait_cycles":16256,"thread_start_us":101,"threads_started":1}
I20260810 18:01:37.352298 2864 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.83.124:37029: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2c754725a56f48c59f4fa09998b76521"
member_type: NON_VOTER
last_known_addr {
host: "127.28.83.124"
port: 37029
}
attrs {
promote: true
}
}
dest_uuid: "e68f257ff67e4efebe5bb38e1fafe09b"
cas_config_opid_index: 5
I20260810 18:01:37.352883 2864 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } }
I20260810 18:01:37.354435 3446 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2c754725a56f48c59f4fa09998b76521 (127.28.83.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.355600 3581 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Refusing update from remote peer e68f257ff67e4efebe5bb38e1fafe09b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 18:01:37.355773 3446 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.83.124:37029
I20260810 18:01:37.355953 3446 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.356060 3446 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:37.356428 3449 master.cc:537] Master@127.28.83.124:37029 shutting down...
I20260810 18:01:37.356792 3446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:01:37.360405 2850 consensus_peers.cc:487] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b -> Peer 2c754725a56f48c59f4fa09998b76521 (127.28.83.124:37029): Couldn't send request to peer 2c754725a56f48c59f4fa09998b76521. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:01:37.362320 3449 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:37.362608 3449 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:37.362696 3449 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: stopping tablet replica
I20260810 18:01:37.364759 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.354593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.0.0.1:56492)
0810 18:01:37.356499 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.356515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.356607 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.360706 (+ 4099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.360709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.360729 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:37.360830 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.360836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.361892 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.361895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.363157 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.363168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.364300 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.364304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.364394 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.364444 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.364522 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.364615 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1820}
I20260810 18:01:37.365679 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.354718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.28.83.125:42411)
0810 18:01:37.354838 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.354843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.360390 (+ 5547us) server_negotiation.cc:407] Connection header received
0810 18:01:37.360463 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.360468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.360540 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.360673 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:37.360975 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.361743 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.363311 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.363943 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.364069 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.364230 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.365040 (+ 810us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.365340 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:01:37.366310 3495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 LEARNER]: Refusing update from remote peer e68f257ff67e4efebe5bb38e1fafe09b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 18:01:37.366995 3637 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:01:37.369829 3449 master.cc:552] Master@127.28.83.124:37029 shutdown complete.
I20260810 18:01:37.369925 3449 master_runner.cc:305] Clearing existing system tablet
I20260810 18:01:37.369777 3495 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2c754725a56f48c59f4fa09998b76521 (127.28.83.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.370388 3449 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:37.370492 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:37.372013 3526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.372159 3526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: This master's current role is: LEARNER
I20260810 18:01:37.372360 3449 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:37.371997 3640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68f257ff67e4efebe5bb38e1fafe09b. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: NON_VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: true } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.373217 3640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: This master's current role is: LEARNER
I20260810 18:01:37.373363 3641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.373620 3449 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:37.373778 3449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:37.373816 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal
uuid: "2c754725a56f48c59f4fa09998b76521"
format_stamp: "Formatted at 2026-08-10 18:01:37 on dist-test-slave-srgv"
server_key: "91c99584a77fcdab0d8d263bdfdfc96a"
server_key_iv: "fd303c67fcaa8c58a2374a1369a4bfe0"
server_key_version: "encryptionkey@0"
I20260810 18:01:37.375594 3449 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:37.377931 3449 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 18:01:37.378036 3449 log.cc:1196] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 18:01:37.378343 3449 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Deleting consensus metadata
I20260810 18:01:37.378535 3449 master_runner.cc:315] Copying system tablet from 127.28.83.126:39527
I20260810 18:01:37.379824 3449 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: tablet copy: Beginning tablet copy session from remote peer at address 127.28.83.126:39527
I20260810 18:01:37.379908 3637 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: attempting to promote NON_VOTER 55c799d9947b47f0a67d47b3053da1ff to VOTER
I20260810 18:01:37.380525 3637 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } }
I20260810 18:01:37.382515 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.382089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.0.0.1:33732)
0810 18:01:37.382343 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.382430 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:37029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260810 18:01:37.383270 3495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 LEARNER]: Refusing update from remote peer e68f257ff67e4efebe5bb38e1fafe09b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 18:01:37.384073 3445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:01:37.384917 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.384578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.0.0.1:33734)
0810 18:01:37.384784 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.384841 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:37029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260810 18:01:37.386294 2850 consensus_peers.cc:487] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b -> Peer 2c754725a56f48c59f4fa09998b76521 (127.28.83.124:37029): Couldn't send request to peer 2c754725a56f48c59f4fa09998b76521. Status: Network error: Client connection negotiation failed: client connection to 127.28.83.124:37029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:01:37.387430 3445 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 55c799d9947b47f0a67d47b3053da1ff (127.28.83.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.389312 3445 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.389446 3445 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:37.389613 3445 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.389694 3445 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:37.390558 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.380535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.28.83.126:39527)
0810 18:01:37.380671 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.380675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.381294 (+ 619us) server_negotiation.cc:407] Connection header received
0810 18:01:37.381533 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.381536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.381587 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.381652 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.382362 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.383137 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.384336 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.384656 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.384732 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.386534 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.386538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.386540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.386568 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.388433 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.389026 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.389032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.389049 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.389142 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.389513 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.389517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.389519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.389770 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.389853 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.390295 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.390421 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:01:37.390575 3495 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 55c799d9947b47f0a67d47b3053da1ff (127.28.83.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.391132 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.380789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50028)
0810 18:01:37.381195 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.381211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.381376 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.381839 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.381842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.382090 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.382228 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.382239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.383283 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.383289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.384135 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.384146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.384898 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.384901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.385007 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.385836 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.385877 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.386368 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.388601 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.388604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.388606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.388898 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.389273 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.389276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.389278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.389375 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.389965 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.389969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.390095 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.390729 (+ 634us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.390891 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":98,"threads_started":1}
I20260810 18:01:37.391477 2886 tablet_copy_service.cc:140] P e68f257ff67e4efebe5bb38e1fafe09b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2c754725a56f48c59f4fa09998b76521 ({username='oryx', principal='oryx/127.28.83.124@KRBTEST.COM'} at 127.0.0.1:50028)
I20260810 18:01:37.391559 2886 tablet_copy_service.cc:161] P e68f257ff67e4efebe5bb38e1fafe09b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2c754725a56f48c59f4fa09998b76521 at {username='oryx', principal='oryx/127.28.83.124@KRBTEST.COM'} at 127.0.0.1:50028: session id = 2c754725a56f48c59f4fa09998b76521-00000000000000000000000000000000
I20260810 18:01:37.392746 2886 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:01:37.393110 3640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.393270 3640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:01:37.394361 3449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 18:01:37.396431 3449 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: tablet copy: Starting download of 0 data blocks...
I20260810 18:01:37.396654 3449 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: tablet copy: Starting download of 1 WAL segments...
I20260810 18:01:37.398185 3449 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:01:37.399955 2886 tablet_copy_service.cc:342] P e68f257ff67e4efebe5bb38e1fafe09b: Request end of tablet copy session 2c754725a56f48c59f4fa09998b76521-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.83.124@KRBTEST.COM'} at 127.0.0.1:50028
I20260810 18:01:37.400063 2886 tablet_copy_service.cc:433] P e68f257ff67e4efebe5bb38e1fafe09b: ending tablet copy session 2c754725a56f48c59f4fa09998b76521-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2c754725a56f48c59f4fa09998b76521
I20260810 18:01:37.407922 3449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:37.409636 3649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:37.413890 3650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:37.414433 3652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:37.414812 3449 server_base.cc:1034] running on GCE node
I20260810 18:01:37.414968 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:37.415270 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:37.418510 3449 hybrid_clock.cc:648] HybridClock initialized: now 1786384897418487 us; error 40 us; skew 500 ppm
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:37.420114 3449 init.cc:377] Logged in from keytab as oryx/127.28.83.124@KRBTEST.COM (short username oryx)
I20260810 18:01:37.420636 3449 webserver.cc:460] Webserver started at http://127.28.83.124:42595/ using document root <none> and password file <none>
I20260810 18:01:37.420867 3449 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:37.420923 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:37.422173 3449 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:37.423194 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.423431 3449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:37.423527 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal
uuid: "2c754725a56f48c59f4fa09998b76521"
format_stamp: "Formatted at 2026-08-10 18:01:37 on dist-test-slave-srgv"
server_key: "91c99584a77fcdab0d8d263bdfdfc96a"
server_key_iv: "fd303c67fcaa8c58a2374a1369a4bfe0"
server_key_version: "encryptionkey@0"
I20260810 18:01:37.423631 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:37.438302 3199 heartbeater.cc:503] Master 127.28.83.126:39527 was elected leader, sending a full tablet report...
I20260810 18:01:37.449618 3449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:37.451757 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:37.460217 3449 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.124:37029
I20260810 18:01:37.460299 3716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.124:37029 every 8 connection(s)
I20260810 18:01:37.460879 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
I20260810 18:01:37.462945 3717 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:01:37.463902 3717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Bootstrap starting.
I20260810 18:01:37.473706 3717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:01:37.474172 3717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Bootstrap complete.
I20260810 18:01:37.474846 3717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } }
I20260810 18:01:37.474995 3717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c754725a56f48c59f4fa09998b76521, State: Initialized, Role: LEARNER
I20260810 18:01:37.475116 3717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } }
I20260810 18:01:37.475656 3720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.475829 3717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:37.475841 3720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: This master's current role is: LEARNER
I20260810 18:01:37.480942 3732 catalog_manager.cc:1259] Loaded cluster ID: bcfc9d0c64304b67b1ea56372f488b35
I20260810 18:01:37.481076 3732 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: loading cluster ID for follower catalog manager: success
I20260810 18:01:37.482723 3732 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: acquiring CA information for follower catalog manager: success
I20260810 18:01:37.483882 3732 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 18:01:37.548996 3636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:37.549355 3636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:37.549465 3636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:37.553776 3636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:37.553874 3636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:37.553974 3636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:37.559913 3636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab
--principal=oryx/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:42123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=38145
--tserver_master_addrs=127.28.83.126:39527,127.28.83.125:42411,127.28.83.124:37029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:37.561095 3636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:37.562258 3636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:37.572642 3737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:37.572651 3740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:37.572693 3738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:37.572850 3636 server_base.cc:1034] running on GCE node
I20260810 18:01:37.573385 3636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:37.574124 3636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:37.575311 3636 hybrid_clock.cc:648] HybridClock initialized: now 1786384897575292 us; error 39 us; skew 500 ppm
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:37.578711 3636 init.cc:377] Logged in from keytab as oryx/127.28.83.65@KRBTEST.COM (short username oryx)
I20260810 18:01:37.579720 3636 webserver.cc:460] Webserver started at http://127.28.83.65:38145/ using document root <none> and password file <none>
I20260810 18:01:37.580259 3636 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:37.580338 3636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:37.584718 3636 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:37.586977 3747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.588251 3636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:37.588402 3636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "63f939c8313248218f4d26b864ae4dce"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "6e2ea94953be2a4244ffa5b7f9005aac"
server_key_iv: "1c8ecf1a27a60974d8393eb325330e7f"
server_key_version: "encryptionkey@0"
I20260810 18:01:37.589082 3636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:37.625666 3636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:37.629374 3636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:37.629628 3636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:37.630378 3636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:37.631356 3636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:37.631407 3636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.631448 3636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:37.631464 3636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.643246 3636 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:42123
I20260810 18:01:37.643364 3864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:42123 every 8 connection(s)
I20260810 18:01:37.644501 3636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:37.652810 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 3636
I20260810 18:01:37.653031 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 3067
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.125@KRBTEST.COM
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.124@KRBTEST.COM
I20260810 18:01:37.661967 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:44715
--local_ip_for_outbound_sockets=127.28.83.66
--tserver_master_addrs=127.28.83.126:39527
--webserver_port=33787
--webserver_interface=127.28.83.66
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab
--principal=oryx/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.83.126:39527,127.28.83.125:42411
--tserver_master_addrs=127.28.83.126:39527,127.28.83.125:42411,127.28.83.124:37029 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:01:37.662096 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.646457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:53147 (local address 127.28.83.124:37029)
0810 18:01:37.646737 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.646742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.647839 (+ 1097us) server_negotiation.cc:407] Connection header received
0810 18:01:37.649016 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.649021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.649082 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.649174 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.650785 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.651864 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.653253 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.653744 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.653852 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.657873 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.657878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.657881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.657911 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.659651 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.660215 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.660219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.660221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.660312 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.660675 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.660680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.660681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.660893 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.661005 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.661718 (+ 713us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.661880 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
I20260810 18:01:37.663079 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.648227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:49879 (local address 127.28.83.125:42411)
0810 18:01:37.648368 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.648371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.648385 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:01:37.649309 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.649312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.649359 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.649495 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.650760 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.651365 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.653327 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.653695 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.653777 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.659248 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.659255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.659261 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.659300 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.661220 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.661790 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.661793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.661795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.661881 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.662208 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.662211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.662212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.662422 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.662484 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.662797 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.662920 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:01:37.663715 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.646896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.28.83.65:53147)
0810 18:01:37.647728 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.647743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.648660 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.649273 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.649282 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.649952 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.650620 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.650629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.652074 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.652077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.653091 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.653115 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.654149 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.654156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.654295 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.655962 (+ 1667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.655982 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.657641 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.659781 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.659788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.659804 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.660080 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.660411 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.660415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.660417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.660572 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.661003 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.661011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.661358 (+ 347us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.661635 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.662026 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":179,"spinlock_wait_cycles":55680,"thread_start_us":160,"threads_started":1}
I20260810 18:01:37.663712 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.647962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.28.83.65:49879)
0810 18:01:37.648246 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.648261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.648660 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.649512 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.649515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.649952 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.650604 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.650619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.651542 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.651545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.653183 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.653194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.653990 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.653994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.654197 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.655012 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.655039 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.659050 (+ 4011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.661352 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.661358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.661363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.661657 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.661999 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.662002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.662004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.662096 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.662544 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.662546 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.662620 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.662728 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.662879 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":291,"thread_start_us":96,"threads_started":1}
I20260810 18:01:37.665585 3865 heartbeater.cc:347] Connected to a master server at 127.28.83.124:37029
I20260810 18:01:37.665835 3865 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:37.666446 3865 heartbeater.cc:511] Master 127.28.83.124:37029 requested a full tablet report, sending...
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:37.667318 3867 heartbeater.cc:347] Connected to a master server at 127.28.83.125:42411
I20260810 18:01:37.667411 3867 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:37.667618 3867 heartbeater.cc:511] Master 127.28.83.125:42411 requested a full tablet report, sending...
I20260810 18:01:37.668316 3677 ts_manager.cc:194] Registered new tserver with Master: 63f939c8313248218f4d26b864ae4dce (127.28.83.65:42123)
I20260810 18:01:37.668848 3485 ts_manager.cc:194] Registered new tserver with Master: 63f939c8313248218f4d26b864ae4dce (127.28.83.65:42123)
I20260810 18:01:37.671156 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.646896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.65:37319)
0810 18:01:37.647501 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.647556 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.648664 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.649274 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.649283 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.649954 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.650620 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.650629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.651757 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.651760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.664154 (+ 12394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.664169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.664951 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.664955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.665051 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.665621 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.665633 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.667210 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.669453 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.669456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.669458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.669691 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.670062 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.670065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.670066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.670184 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.670703 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.670706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.670786 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.670865 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.670983 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":166,"spinlock_wait_cycles":20608,"thread_start_us":148,"threads_started":1}
I20260810 18:01:37.671378 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.646438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:37319 (local address 127.28.83.126:39527)
0810 18:01:37.646580 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.646585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.647686 (+ 1101us) server_negotiation.cc:407] Connection header received
0810 18:01:37.648902 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.648905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.648973 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.649060 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.650786 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.651610 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.664325 (+ 12715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.664653 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.664746 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.667397 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.667401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.667404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.667437 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.669266 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.669828 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.669834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.669838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.669947 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.670301 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.670308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.670313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.670591 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.670714 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.670978 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.671145 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:01:37.672293 3866 heartbeater.cc:347] Connected to a master server at 127.28.83.126:39527
I20260810 18:01:37.672415 3866 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:37.672610 3866 heartbeater.cc:511] Master 127.28.83.126:39527 requested a full tablet report, sending...
I20260810 18:01:37.673589 2864 ts_manager.cc:194] Re-registered known tserver with Master: 63f939c8313248218f4d26b864ae4dce (127.28.83.65:42123)
I20260810 18:01:37.674106 3338 heartbeater.cc:503] Master 127.28.83.126:39527 was elected leader, sending a full tablet report...
I20260810 18:01:37.674500 2864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:37319
W20260810 18:01:37.782027 3873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:37.782380 3873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:37.782450 3873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 18:01:37.783440 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.779586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.28.83.124:37029)
0810 18:01:37.779733 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.779738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.779790 (+ 52us) server_negotiation.cc:407] Connection header received
0810 18:01:37.779941 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.779944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.780013 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.780136 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:37.780370 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.781194 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.782241 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.782924 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.783040 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.783206 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.783240 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.783308 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:01:37.783444 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.779504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.0.0.1:33746)
0810 18:01:37.779722 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.779736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.779832 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.780146 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.780148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.780161 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:37.780257 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.780261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.781304 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.781306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.782098 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.782109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.783049 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.783052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.783119 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.783140 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.783192 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.783301 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260810 18:01:37.784152 3688 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 18:01:37.784582 3720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68f257ff67e4efebe5bb38e1fafe09b. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:37.784765 3720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: This master's current role is: LEARNER
W20260810 18:01:37.787482 3873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:37.787542 3873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:37.787663 3873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:37.793329 3873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab
--principal=oryx/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:44715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=33787
--tserver_master_addrs=127.28.83.126:39527,127.28.83.125:42411,127.28.83.124:37029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:37.794601 3873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:37.795746 3873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:37.805639 3881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:37.805887 3879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:37.805722 3873 server_base.cc:1034] running on GCE node
W20260810 18:01:37.806234 3878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:37.806721 3873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:37.807284 3873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:37.808437 3873 hybrid_clock.cc:648] HybridClock initialized: now 1786384897808404 us; error 47 us; skew 500 ppm
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:37.811414 3873 init.cc:377] Logged in from keytab as oryx/127.28.83.66@KRBTEST.COM (short username oryx)
I20260810 18:01:37.812285 3873 webserver.cc:460] Webserver started at http://127.28.83.66:33787/ using document root <none> and password file <none>
I20260810 18:01:37.812855 3873 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:37.812975 3873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:37.816783 3873 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:37.818845 3888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.819998 3873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 18:01:37.820159 3873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "6e032e99470087278216108151c3dcc5"
server_key_iv: "8a9d3e6abd1a974d5d38bdae7c9ce6f0"
server_key_version: "encryptionkey@0"
I20260810 18:01:37.820714 3873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:37.838433 3873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:37.841518 3873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:37.841776 3873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:37.842404 3873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:37.843392 3873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:37.843477 3873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.843552 3873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:37.843600 3873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:37.852838 3873 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:44715
I20260810 18:01:37.852870 4007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:44715 every 8 connection(s)
I20260810 18:01:37.853981 3873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:37.860630 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 3873
I20260810 18:01:37.860819 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 3204
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.124@KRBTEST.COM
Aug 10 18:01:37 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.125@KRBTEST.COM
I20260810 18:01:37.871922 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.855842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:51299 (local address 127.28.83.126:39527)
0810 18:01:37.855979 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.855984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.857090 (+ 1106us) server_negotiation.cc:407] Connection header received
0810 18:01:37.858368 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.858371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.858417 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.858511 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.859745 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.860280 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.862264 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.862621 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.862696 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.867014 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.867018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.867022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.867053 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.869092 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.869874 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.869880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.869883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.869978 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.870496 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.870500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.870502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.870753 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.870845 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.871595 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.871751 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:01:37.871937 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:35521
--local_ip_for_outbound_sockets=127.28.83.67
--tserver_master_addrs=127.28.83.126:39527
--webserver_port=35893
--webserver_interface=127.28.83.67
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab
--principal=oryx/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.83.126:39527,127.28.83.125:42411
--tserver_master_addrs=127.28.83.126:39527,127.28.83.125:42411,127.28.83.124:37029 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:01:37.872068 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.857450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:35873 (local address 127.28.83.125:42411)
0810 18:01:37.857577 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.857592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.857695 (+ 103us) server_negotiation.cc:407] Connection header received
0810 18:01:37.858654 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.858658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.858726 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.858815 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.859754 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.860569 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.863672 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.864061 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.864162 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.868287 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.868294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.868300 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.868337 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.870173 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.870701 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.870704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.870707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.870797 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.871083 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.871087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.871088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.871359 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.871459 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.871804 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.871920 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:01:37.872169 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.855915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:49593 (local address 127.28.83.124:37029)
0810 18:01:37.856043 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:37.856047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:37.856998 (+ 951us) server_negotiation.cc:407] Connection header received
0810 18:01:37.858095 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:37.858099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:37.858170 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:37.858240 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:37.859755 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.860569 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.862562 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.862897 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.862974 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.867024 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:37.867030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:37.867037 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:37.867072 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:37.869516 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.870047 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.870054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.870059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.870151 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:37.870517 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:37.870523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:37.870528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:37.870777 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:37.870859 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:37.871832 (+ 973us) server_negotiation.cc:299] Negotiation successful
0810 18:01:37.871966 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:01:37.873680 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.857307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.28.83.66:35873)
0810 18:01:37.857583 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.857597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.857896 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.858854 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.858856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.859044 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.859585 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.859601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.860721 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.860723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.863515 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.863526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.864921 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.864926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.865020 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.865696 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.865711 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.868076 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.870290 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.870293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.870295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.870579 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.870907 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.870910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.870912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.870995 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.871462 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.871468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.871613 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.873005 (+ 1392us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.873144 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":1700,"spinlock_wait_cycles":20224,"thread_start_us":109,"threads_started":1}
I20260810 18:01:37.873689 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.856300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.66:51299)
0810 18:01:37.856987 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.857002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.857878 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.858528 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.858531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.859004 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.859584 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.859599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.860498 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.860504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.862118 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.862128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.862893 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.862897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.863111 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.863982 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.864004 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.866794 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.869268 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.869278 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.869302 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.869745 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.870112 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.870115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.870119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.870377 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.870916 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.870923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.871297 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.872095 (+ 798us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.872532 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":450,"thread_start_us":142,"threads_started":1}
I20260810 18:01:37.874029 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:37.856300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.28.83.66:49593)
0810 18:01:37.856833 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:37.856886 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:37.857892 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:37.858437 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:37.858444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:37.858998 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:37.859592 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.859601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.860841 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.860844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.862437 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:37.862446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:37.863129 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:37.863132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:37.863218 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:37.864313 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:37.864330 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:37.866852 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:37.869644 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.869651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.869656 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.869928 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.870281 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:37.870286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:37.870291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:37.870404 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:37.870921 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:37.870926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:37.871616 (+ 690us) client_negotiation.cc:769] Sending connection context
0810 18:01:37.872437 (+ 821us) client_negotiation.cc:240] Negotiation successful
0810 18:01:37.872622 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":1250,"spinlock_wait_cycles":2816,"thread_start_us":126,"threads_started":1}
I20260810 18:01:37.875195 4009 heartbeater.cc:347] Connected to a master server at 127.28.83.126:39527
I20260810 18:01:37.875233 4008 heartbeater.cc:347] Connected to a master server at 127.28.83.124:37029
I20260810 18:01:37.875264 4010 heartbeater.cc:347] Connected to a master server at 127.28.83.125:42411
I20260810 18:01:37.875444 4010 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:37.875444 4009 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:37.875444 4008 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:37.876091 4010 heartbeater.cc:511] Master 127.28.83.125:42411 requested a full tablet report, sending...
I20260810 18:01:37.876087 4009 heartbeater.cc:511] Master 127.28.83.126:39527 requested a full tablet report, sending...
I20260810 18:01:37.876087 4008 heartbeater.cc:511] Master 127.28.83.124:37029 requested a full tablet report, sending...
I20260810 18:01:37.877312 2863 ts_manager.cc:194] Re-registered known tserver with Master: 7fa7b864070f42dbaf8af4a9ba72a0a8 (127.28.83.66:44715)
I20260810 18:01:37.877452 3677 ts_manager.cc:194] Registered new tserver with Master: 7fa7b864070f42dbaf8af4a9ba72a0a8 (127.28.83.66:44715)
I20260810 18:01:37.877839 3485 ts_manager.cc:194] Registered new tserver with Master: 7fa7b864070f42dbaf8af4a9ba72a0a8 (127.28.83.66:44715)
I20260810 18:01:37.879025 2863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:51299
W20260810 18:01:37.998139 4015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:37.998420 4015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:37.998466 4015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:38.002749 4015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:38.002810 4015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:38.002871 4015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:38.008206 4015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab
--principal=oryx/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:35521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=35893
--tserver_master_addrs=127.28.83.126:39527,127.28.83.125:42411,127.28.83.124:37029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:38.009236 4015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:38.010273 4015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:38.019997 4021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:38.020085 4020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:38.020175 4015 server_base.cc:1034] running on GCE node
W20260810 18:01:38.020009 4023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:38.020694 4015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:38.021268 4015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:38.022428 4015 hybrid_clock.cc:648] HybridClock initialized: now 1786384898022412 us; error 30 us; skew 500 ppm
Aug 10 18:01:38 dist-test-slave-srgv krb5kdc[2818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:38.025547 4015 init.cc:377] Logged in from keytab as oryx/127.28.83.67@KRBTEST.COM (short username oryx)
I20260810 18:01:38.026435 4015 webserver.cc:460] Webserver started at http://127.28.83.67:35893/ using document root <none> and password file <none>
I20260810 18:01:38.027000 4015 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:38.027081 4015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:38.030871 4015 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 18:01:38.033010 4030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:38.034281 4015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:38.034400 4015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "b48a1c3a95094db1a8be66645ecfccb7"
format_stamp: "Formatted at 2026-08-10 18:01:36 on dist-test-slave-srgv"
server_key: "48179fb9aec99d017fe4d6ebdb9a4eb3"
server_key_iv: "560c426dc2f37cc195f287137186a914"
server_key_version: "encryptionkey@0"
I20260810 18:01:38.034955 4015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:38.048228 4015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:38.051371 4015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:38.051574 4015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:38.052217 4015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:38.053099 4015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:38.053148 4015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:38.053213 4015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:38.053244 4015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:38.062516 4015 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:35521
I20260810 18:01:38.062541 4145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:35521 every 8 connection(s)
I20260810 18:01:38.063594 4015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:38.071123 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 4015
Aug 10 18:01:38 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.125@KRBTEST.COM
Aug 10 18:01:38 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
Aug 10 18:01:38 dist-test-slave-srgv krb5kdc[2818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.124@KRBTEST.COM
I20260810 18:01:38.081379 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.067574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:57813 (local address 127.28.83.125:42411)
0810 18:01:38.067763 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.067767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.067784 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:01:38.068277 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.068280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.068337 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.068424 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.070048 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.070908 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.072196 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.072508 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.072571 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.076558 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.076565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.076571 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.076606 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.078528 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.079196 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.079214 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.079219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.079312 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.080023 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.080028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.080033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.080289 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.080379 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.081077 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.081216 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 18:01:38.081379 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.065629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:32781 (local address 127.28.83.124:37029)
0810 18:01:38.065770 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.065775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.067088 (+ 1313us) server_negotiation.cc:407] Connection header received
0810 18:01:38.068296 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.068300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.068360 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.068415 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.070117 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.070912 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.074161 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.074556 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.074657 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.077473 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.077478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.077482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.077518 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.079265 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.080075 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.080081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.080084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.080166 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.080478 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.080481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.080482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.080684 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.080779 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.081094 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.081210 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:01:38.083112 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.066160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.28.83.67:32781)
0810 18:01:38.066879 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:38.066959 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:38.068042 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:38.068633 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:38.068643 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:38.069239 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:38.069941 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.069951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.073060 (+ 3109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.073063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.073999 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.074013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.074854 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.074861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.074976 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.075818 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:38.075834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:38.077254 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:38.079415 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:38.079420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:38.079422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:38.079677 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:38.080278 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:38.080280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:38.080282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:38.080373 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:38.080837 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:38.080839 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:38.080913 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 18:01:38.081566 (+ 653us) client_negotiation.cc:240] Negotiation successful
0810 18:01:38.082129 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":433,"spinlock_wait_cycles":29568,"thread_start_us":133,"threads_started":1}
I20260810 18:01:38.083789 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.067331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.28.83.67:57813)
0810 18:01:38.067652 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:38.067668 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:38.068050 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:38.068645 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:38.068648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:38.069204 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:38.069894 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.069910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.071056 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.071062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.072015 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.072055 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.072746 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.072748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.072883 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.073615 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:38.073641 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:38.076358 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:38.078681 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:38.078688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:38.078708 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:38.079068 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:38.079437 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:38.079439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:38.079441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:38.079937 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:38.080435 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:38.080444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:38.080784 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 18:01:38.081566 (+ 782us) client_negotiation.cc:240] Negotiation successful
0810 18:01:38.082041 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":73,"thread_start_us":102,"threads_started":1}
I20260810 18:01:38.084070 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.065629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:40937 (local address 127.28.83.126:39527)
0810 18:01:38.065770 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.065775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.067118 (+ 1343us) server_negotiation.cc:407] Connection header received
0810 18:01:38.068297 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.068300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.068360 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.068415 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.070113 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.070646 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.072342 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.072641 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.072727 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.079619 (+ 6892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.079623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.079626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.079655 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.081490 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.082400 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.082404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.082405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.082492 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.082826 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.082829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.082831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.083031 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.083133 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.083757 (+ 624us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.083888 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:01:38.084990 4149 heartbeater.cc:347] Connected to a master server at 127.28.83.125:42411
I20260810 18:01:38.085114 4146 heartbeater.cc:347] Connected to a master server at 127.28.83.124:37029
I20260810 18:01:38.085253 4146 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:38.085845 4146 heartbeater.cc:511] Master 127.28.83.124:37029 requested a full tablet report, sending...
I20260810 18:01:38.085963 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.066160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.67:40937)
0810 18:01:38.066991 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:38.067008 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:38.068042 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:38.068633 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:38.068643 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:38.069239 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:38.069941 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.069951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.070796 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.070798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.072181 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.072196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.072897 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.072900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.073015 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.073902 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:38.073917 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:38.077740 (+ 3823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:38.081998 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:38.082002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:38.082004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:38.082280 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:38.082601 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:38.082603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:38.082606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:38.082723 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:38.083443 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:38.083449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:38.083553 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 18:01:38.085566 (+ 2013us) client_negotiation.cc:240] Negotiation successful
0810 18:01:38.085725 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":190,"spinlock_wait_cycles":49152,"thread_start_us":161,"threads_started":1}
I20260810 18:01:38.085255 4149 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:38.086607 4149 heartbeater.cc:511] Master 127.28.83.125:42411 requested a full tablet report, sending...
I20260810 18:01:38.087114 4147 heartbeater.cc:347] Connected to a master server at 127.28.83.126:39527
I20260810 18:01:38.087199 4147 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:38.087342 4147 heartbeater.cc:511] Master 127.28.83.126:39527 requested a full tablet report, sending...
I20260810 18:01:38.087488 3677 ts_manager.cc:194] Registered new tserver with Master: b48a1c3a95094db1a8be66645ecfccb7 (127.28.83.67:35521)
I20260810 18:01:38.087909 2863 ts_manager.cc:194] Re-registered known tserver with Master: b48a1c3a95094db1a8be66645ecfccb7 (127.28.83.67:35521)
I20260810 18:01:38.088657 2863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:40937
I20260810 18:01:38.089535 3485 ts_manager.cc:194] Registered new tserver with Master: b48a1c3a95094db1a8be66645ecfccb7 (127.28.83.67:35521)
I20260810 18:01:38.095727 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.075559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.28.83.125:42411)
0810 18:01:38.075900 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.075907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.078076 (+ 2169us) server_negotiation.cc:407] Connection header received
0810 18:01:38.078121 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.078124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.078221 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.078305 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.079282 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.081381 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.082724 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.083061 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.083159 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.091391 (+ 8232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.091397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.091400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.091428 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.093295 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.093920 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.093926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.093929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.094016 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.094328 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.094331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.094333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.094597 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.094731 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.095402 (+ 671us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.095535 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":118,"threads_started":1}
I20260810 18:01:38.096000 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.073257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.28.83.126:39527)
0810 18:01:38.074072 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.074076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.074092 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:01:38.074181 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.074184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.074245 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.074343 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.075324 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.076155 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.078304 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.078971 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.079092 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.091143 (+ 12051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.091147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.091150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.091177 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.093444 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.094132 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.094138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.094139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.094238 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.094535 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.094539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.094540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.094824 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.094942 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.095705 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.095837 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":102,"threads_started":1}
I20260810 18:01:38.096025 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.076952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.28.83.124:37029)
0810 18:01:38.077657 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.077662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.083574 (+ 5912us) server_negotiation.cc:407] Connection header received
0810 18:01:38.083630 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.083636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.083696 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.083773 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.085360 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.086161 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.087572 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.087890 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.087972 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.091218 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.091223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.091225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.091254 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.093668 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.094343 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.094346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.094348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.094430 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.094725 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.094728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.094730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.094962 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.095098 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.095482 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.095603 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":97,"threads_started":1}
I20260810 18:01:38.099089 2863 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:38.102370 2863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:38.104884 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: NON_VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: true } } }
I20260810 18:01:38.105031 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:38.108421 4166 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: attempting to promote NON_VOTER 2c754725a56f48c59f4fa09998b76521 to VOTER
I20260810 18:01:38.108779 4166 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } }
I20260810 18:01:38.112007 3688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 LEARNER]: Refusing update from remote peer e68f257ff67e4efebe5bb38e1fafe09b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 18:01:38.112037 3495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Refusing update from remote peer e68f257ff67e4efebe5bb38e1fafe09b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 18:01:38.112378 4165 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 18:01:38.112519 3637 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 18:01:38.114711 4166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2c754725a56f48c59f4fa09998b76521 (127.28.83.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:38.115118 3495 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2c754725a56f48c59f4fa09998b76521 (127.28.83.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:38.115159 3688 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2c754725a56f48c59f4fa09998b76521 (127.28.83.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:38.116809 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:38.116931 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:38.117076 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:38.117048 4165 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:38.117174 4165 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:38.117264 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:38.117399 3720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:38.117559 3720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:01:38.118288 4176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:38.118384 4176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:01:38.124577 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.118374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:35521 (local address 127.0.0.1:43288)
0810 18:01:38.118730 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:38.118746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:38.118829 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:38.120414 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:38.120417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:38.120437 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:38.120567 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.120574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.121922 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.121926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.123000 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.123007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.124140 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.124144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.124253 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.124275 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:01:38.124350 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:01:38.124418 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":61,"spinlock_wait_cycles":384}
I20260810 18:01:38.124651 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.118723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.28.83.67:35521)
0810 18:01:38.119960 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.119973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.119996 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:01:38.120070 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.120086 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.120283 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.120438 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:38.120679 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.121804 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.123106 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.123984 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.124140 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.124274 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.124376 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.124467 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":112,"threads_started":1}
I20260810 18:01:38.124773 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.119233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.28.83.65:42123)
0810 18:01:38.119591 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.119609 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.119680 (+ 71us) server_negotiation.cc:407] Connection header received
0810 18:01:38.119805 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.119813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.120092 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.120232 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:38.120697 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.121454 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.122814 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.123651 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.123815 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.123955 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.124522 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.124635 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":79,"threads_started":1}
I20260810 18:01:38.124433 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.118954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42123 (local address 127.0.0.1:54408)
0810 18:01:38.119581 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:38.119595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:38.119690 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:38.120380 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:38.120385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:38.120407 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:38.120567 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.120574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.121548 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.121554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.122673 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.122683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.124006 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.124009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.124147 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.124167 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:01:38.124242 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:01:38.124322 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":36,"spinlock_wait_cycles":1920,"thread_start_us":130,"threads_started":1}
I20260810 18:01:38.125299 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.118742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:44715 (local address 127.0.0.1:38864)
0810 18:01:38.119220 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:38.119234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:38.119328 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:38.120202 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:38.120205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:38.120223 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:38.120339 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.120349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.121837 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.121843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.122705 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.122715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.123954 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.123957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.124073 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.125024 (+ 951us) client_negotiation.cc:769] Sending connection context
0810 18:01:38.125110 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 18:01:38.125180 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":952,"thread_start_us":109,"threads_started":1}
I20260810 18:01:38.125829 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.119175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.28.83.66:44715)
0810 18:01:38.119805 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.119817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.119835 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:01:38.119898 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.119902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.120050 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.120230 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:38.120479 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.121703 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.122868 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.123661 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.123807 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.124224 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.125580 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.125687 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":90,"threads_started":1}
I20260810 18:01:38.127799 3797 tablet_service.cc:1467] Processing CreateTablet for tablet 49cf7d0552f84e14a2828228e6f48d2c (DEFAULT_TABLE table=test-table [id=c0c43f97b8ce4d47a7447a9102fea84f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:38.127796 4080 tablet_service.cc:1467] Processing CreateTablet for tablet 49cf7d0552f84e14a2828228e6f48d2c (DEFAULT_TABLE table=test-table [id=c0c43f97b8ce4d47a7447a9102fea84f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:38.128921 3797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cf7d0552f84e14a2828228e6f48d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:38.128921 4080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cf7d0552f84e14a2828228e6f48d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:38.135860 3938 tablet_service.cc:1467] Processing CreateTablet for tablet 49cf7d0552f84e14a2828228e6f48d2c (DEFAULT_TABLE table=test-table [id=c0c43f97b8ce4d47a7447a9102fea84f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:38.136799 3938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cf7d0552f84e14a2828228e6f48d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:38.137365 4183 tablet_bootstrap.cc:492] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: Bootstrap starting.
I20260810 18:01:38.139644 4184 tablet_bootstrap.cc:492] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: Bootstrap starting.
I20260810 18:01:38.140156 4183 tablet_bootstrap.cc:654] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:38.141096 4183 log.cc:824] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:38.141754 4184 tablet_bootstrap.cc:654] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:38.142549 4184 log.cc:824] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:38.143383 4183 tablet_bootstrap.cc:492] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: No bootstrap required, opened a new log
I20260810 18:01:38.143575 4183 ts_tablet_manager.cc:1397] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260810 18:01:38.145496 4187 tablet_bootstrap.cc:492] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: Bootstrap starting.
I20260810 18:01:38.146968 4184 tablet_bootstrap.cc:492] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: No bootstrap required, opened a new log
I20260810 18:01:38.147173 4184 ts_tablet_manager.cc:1397] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 18:01:38.148595 4183 raft_consensus.cc:348] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } }
I20260810 18:01:38.148813 4183 raft_consensus.cc:374] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:38.148883 4183 raft_consensus.cc:729] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f939c8313248218f4d26b864ae4dce, State: Initialized, Role: FOLLOWER
I20260810 18:01:38.149341 4183 consensus_queue.cc:260] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } }
I20260810 18:01:38.150219 3866 heartbeater.cc:503] Master 127.28.83.126:39527 was elected leader, sending a full tablet report...
I20260810 18:01:38.151784 4187 tablet_bootstrap.cc:654] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:38.153245 4187 log.cc:824] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:38.154863 4183 ts_tablet_manager.cc:1428] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260810 18:01:38.155241 4184 raft_consensus.cc:348] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } }
I20260810 18:01:38.155776 4184 raft_consensus.cc:374] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:38.155841 4184 raft_consensus.cc:729] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48a1c3a95094db1a8be66645ecfccb7, State: Initialized, Role: FOLLOWER
I20260810 18:01:38.155938 4187 tablet_bootstrap.cc:492] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: No bootstrap required, opened a new log
I20260810 18:01:38.156142 4187 ts_tablet_manager.cc:1397] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260810 18:01:38.156347 4184 consensus_queue.cc:260] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } }
I20260810 18:01:38.157538 4184 ts_tablet_manager.cc:1428] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260810 18:01:38.157840 4147 heartbeater.cc:503] Master 127.28.83.126:39527 was elected leader, sending a full tablet report...
I20260810 18:01:38.162878 4187 raft_consensus.cc:348] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } }
I20260810 18:01:38.163154 4187 raft_consensus.cc:374] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:38.163220 4187 raft_consensus.cc:729] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa7b864070f42dbaf8af4a9ba72a0a8, State: Initialized, Role: FOLLOWER
I20260810 18:01:38.163635 4187 consensus_queue.cc:260] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } }
I20260810 18:01:38.164757 4187 ts_tablet_manager.cc:1428] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260810 18:01:38.164776 4009 heartbeater.cc:503] Master 127.28.83.126:39527 was elected leader, sending a full tablet report...
I20260810 18:01:38.193763 4188 raft_consensus.cc:482] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:38.194082 4188 raft_consensus.cc:504] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } }
I20260810 18:01:38.195312 4188 leader_election.cc:288] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b48a1c3a95094db1a8be66645ecfccb7 (127.28.83.67:35521), 7fa7b864070f42dbaf8af4a9ba72a0a8 (127.28.83.66:44715)
I20260810 18:01:38.199460 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.196191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:48097 (local address 127.28.83.66:44715)
0810 18:01:38.196328 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.196331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.196345 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:01:38.196425 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.196427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.196496 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.196574 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:38.196853 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.197342 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.198413 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.198832 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.198923 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.198961 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.199215 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.199313 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:01:38.199482 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.195993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:44715 (local address 127.28.83.65:48097)
0810 18:01:38.196226 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:38.196241 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:38.196340 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:38.196604 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:38.196609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:38.196640 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:38.196742 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.196748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.197469 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.197475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.198287 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.198296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.198949 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.198953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.199069 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.199096 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:01:38.199171 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:01:38.199258 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260810 18:01:38.199697 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.195683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:59135 (local address 127.28.83.67:35521)
0810 18:01:38.195825 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.195828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.195975 (+ 147us) server_negotiation.cc:407] Connection header received
0810 18:01:38.196323 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.196327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.196385 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.196464 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:38.196761 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.197375 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.198478 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.198986 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.199103 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.199167 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.199436 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.199564 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:01:38.200037 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.195596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:35521 (local address 127.28.83.65:59135)
0810 18:01:38.195860 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:38.195877 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:38.195993 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:38.196497 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:38.196501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:38.196522 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:38.196639 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.196649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.197541 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.197544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.198321 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:38.198330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.199113 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.199116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:38.199301 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.199324 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:01:38.199400 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:01:38.199460 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
I20260810 18:01:38.200218 3961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cf7d0552f84e14a2828228e6f48d2c" candidate_uuid: "63f939c8313248218f4d26b864ae4dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" is_pre_election: true
I20260810 18:01:38.200603 3961 raft_consensus.cc:2454] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f939c8313248218f4d26b864ae4dce in term 0.
I20260810 18:01:38.200721 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cf7d0552f84e14a2828228e6f48d2c" candidate_uuid: "63f939c8313248218f4d26b864ae4dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48a1c3a95094db1a8be66645ecfccb7" is_pre_election: true
I20260810 18:01:38.201005 4100 raft_consensus.cc:2454] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f939c8313248218f4d26b864ae4dce in term 0.
I20260810 18:01:38.201309 3750 leader_election.cc:302] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f939c8313248218f4d26b864ae4dce, 7fa7b864070f42dbaf8af4a9ba72a0a8; no voters:
I20260810 18:01:38.201644 4188 raft_consensus.cc:2781] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:38.201733 4188 raft_consensus.cc:482] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:38.201781 4188 raft_consensus.cc:3037] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:38.203380 4188 raft_consensus.cc:504] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } }
I20260810 18:01:38.203759 4188 leader_election.cc:288] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [CANDIDATE]: Term 1 election: Requested vote from peers b48a1c3a95094db1a8be66645ecfccb7 (127.28.83.67:35521), 7fa7b864070f42dbaf8af4a9ba72a0a8 (127.28.83.66:44715)
I20260810 18:01:38.204182 3961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cf7d0552f84e14a2828228e6f48d2c" candidate_uuid: "63f939c8313248218f4d26b864ae4dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8"
I20260810 18:01:38.204221 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cf7d0552f84e14a2828228e6f48d2c" candidate_uuid: "63f939c8313248218f4d26b864ae4dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48a1c3a95094db1a8be66645ecfccb7"
I20260810 18:01:38.204316 3961 raft_consensus.cc:3037] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:38.204347 4100 raft_consensus.cc:3037] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:38.205912 4100 raft_consensus.cc:2454] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f939c8313248218f4d26b864ae4dce in term 1.
I20260810 18:01:38.206040 3961 raft_consensus.cc:2454] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f939c8313248218f4d26b864ae4dce in term 1.
I20260810 18:01:38.206189 3750 leader_election.cc:302] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f939c8313248218f4d26b864ae4dce, b48a1c3a95094db1a8be66645ecfccb7; no voters:
I20260810 18:01:38.206391 4188 raft_consensus.cc:2781] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:38.206602 4188 raft_consensus.cc:686] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 1 LEADER]: Becoming Leader. State: Replica: 63f939c8313248218f4d26b864ae4dce, State: Running, Role: LEADER
I20260810 18:01:38.206941 4188 consensus_queue.cc:237] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } }
I20260810 18:01:38.210350 2863 catalog_manager.cc:5495] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce reported cstate change: term changed from 0 to 1, leader changed from <none> to 63f939c8313248218f4d26b864ae4dce (127.28.83.65). New cstate: current_term: 1 leader_uuid: "63f939c8313248218f4d26b864ae4dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } health_report { overall_health: HEALTHY } } }
I20260810 18:01:38.229331 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.225851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.28.83.65:42123)
0810 18:01:38.225983 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.225987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.226000 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:01:38.226143 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.226145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.226199 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.226287 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:38.226572 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.227134 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.227934 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.228152 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.228235 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.228566 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:38.228962 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:38.229052 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.229111 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.229160 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:01:38.235765 4100 raft_consensus.cc:1261] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [term 1 FOLLOWER]: Refusing update from remote peer 63f939c8313248218f4d26b864ae4dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:38.235765 3961 raft_consensus.cc:1261] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 1 FOLLOWER]: Refusing update from remote peer 63f939c8313248218f4d26b864ae4dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:38.236680 4194 consensus_queue.cc:1035] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:38.236902 4188 consensus_queue.cc:1035] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:38.247836 4198 mvcc.cc:205] Tried to move back new op lower bound from 7317032543165333504 to 7317032543058325504. Current Snapshot: MvccSnapshot[applied={T|T < 7317032543165333504}]
I20260810 18:01:38.250437 4197 mvcc.cc:205] Tried to move back new op lower bound from 7317032543165333504 to 7317032543058325504. Current Snapshot: MvccSnapshot[applied={T|T < 7317032543165333504 or (T in {7317032543165333504})}]
I20260810 18:01:38.252415 4199 mvcc.cc:205] Tried to move back new op lower bound from 7317032543165333504 to 7317032543058325504. Current Snapshot: MvccSnapshot[applied={T|T < 7317032543165333504}]
I20260810 18:01:38.256343 2864 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:38.256523 2864 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:38.260963 2864 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e68f257ff67e4efebe5bb38e1fafe09b: Sending DeleteTablet for 3 replicas of tablet 49cf7d0552f84e14a2828228e6f48d2c
I20260810 18:01:38.261880 4080 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cf7d0552f84e14a2828228e6f48d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:38 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:43288
I20260810 18:01:38.262110 3938 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cf7d0552f84e14a2828228e6f48d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:38 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:38864
I20260810 18:01:38.262456 4206 tablet_replica.cc:330] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: stopping tablet replica
I20260810 18:01:38.262817 4206 raft_consensus.cc:2229] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:38.262773 3797 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cf7d0552f84e14a2828228e6f48d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:38 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:54408
I20260810 18:01:38.263147 4206 raft_consensus.cc:2258] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:38.263244 4208 tablet_replica.cc:330] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: stopping tablet replica
I20260810 18:01:38.263535 4208 raft_consensus.cc:2229] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:38.263875 4208 raft_consensus.cc:2258] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:38.265223 4208 ts_tablet_manager.cc:1905] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:38.265039 4206 ts_tablet_manager.cc:1905] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:38.266467 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 2831
I20260810 18:01:38.269992 4208 ts_tablet_manager.cc:1918] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:38.270102 4208 log.cc:1196] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/49cf7d0552f84e14a2828228e6f48d2c
I20260810 18:01:38.270529 4208 ts_tablet_manager.cc:1939] T 49cf7d0552f84e14a2828228e6f48d2c P 63f939c8313248218f4d26b864ae4dce: Deleting consensus metadata
I20260810 18:01:38.272204 4206 ts_tablet_manager.cc:1918] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:38.272322 4206 log.cc:1196] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/49cf7d0552f84e14a2828228e6f48d2c
I20260810 18:01:38.272746 4206 ts_tablet_manager.cc:1939] T 49cf7d0552f84e14a2828228e6f48d2c P b48a1c3a95094db1a8be66645ecfccb7: Deleting consensus metadata
W20260810 18:01:38.275985 4032 connection.cc:535] server connection from 127.0.0.1:43288 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 18:01:38.276013 3751 connection.cc:535] server connection from 127.0.0.1:54408 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 18:01:38.279738 4207 tablet_replica.cc:330] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: stopping tablet replica
I20260810 18:01:38.280189 4207 raft_consensus.cc:2229] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:38.280579 4207 raft_consensus.cc:2258] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:38.282572 4207 ts_tablet_manager.cc:1905] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:38.285612 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.265700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.28.83.125:42411)
0810 18:01:38.265833 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.265837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.270715 (+ 4878us) server_negotiation.cc:407] Connection header received
0810 18:01:38.270930 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.270935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.271001 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.271058 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.274873 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.275833 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.277278 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.277701 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.277827 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.280030 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.280034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.280036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.280068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.282233 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.282983 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.282986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.282989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.283125 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.283526 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.283529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.283531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.283811 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.283939 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.285274 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.285446 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:01:38.286036 4207 ts_tablet_manager.cc:1918] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:38.286175 4207 log.cc:1196] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/49cf7d0552f84e14a2828228e6f48d2c
I20260810 18:01:38.286538 4207 ts_tablet_manager.cc:1939] T 49cf7d0552f84e14a2828228e6f48d2c P 7fa7b864070f42dbaf8af4a9ba72a0a8: Deleting consensus metadata
W20260810 18:01:38.288043 3892 connection.cc:441] server connection from 127.0.0.1:38864 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38864 (request call id 1) could send its response
I20260810 18:01:38.295550 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.265170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.28.83.124:37029)
0810 18:01:38.265289 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.265293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.284958 (+ 19665us) server_negotiation.cc:407] Connection header received
0810 18:01:38.285004 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.285007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.285059 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.285129 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.286997 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.287929 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.289182 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.289642 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.289747 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.291408 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.291411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.291414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.291441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.293308 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.293857 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.293860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.293862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.293947 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.294349 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.294352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.294354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.294574 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.294692 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.295242 (+ 550us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.295370 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":768}
I20260810 18:01:38.296877 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.281116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.28.83.124:37029)
0810 18:01:38.281267 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.281271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.282697 (+ 1426us) server_negotiation.cc:407] Connection header received
0810 18:01:38.282885 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.282888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.282942 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.282996 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.284947 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.285820 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.287968 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.288294 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.288372 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.290752 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.290756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.290758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.290785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.293687 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.294550 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.294554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.294557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.294673 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.295007 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.295012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.295015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.295228 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.295337 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.296521 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.296693 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 18:01:38.299212 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:38.280565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.28.83.125:42411)
0810 18:01:38.280711 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:38.280715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:38.281989 (+ 1274us) server_negotiation.cc:407] Connection header received
0810 18:01:38.282164 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:38.282167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:38.282216 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:38.282269 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:38.290579 (+ 8310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.291451 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.292779 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:38.293173 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:38.293286 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:38.295130 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:38.295135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:38.295138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:38.295164 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:38.297053 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.297890 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.297894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.297896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.297982 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:38.298291 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:38.298294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:38.298296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:38.298534 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:38.298660 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:38.298916 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 18:01:38.299038 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 18:01:39.251343 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.250812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.65:34995)
0810 18:01:39.251094 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.251222 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":104,"threads_started":1}
W20260810 18:01:39.251742 3866 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:39527 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
I20260810 18:01:39.255888 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.255343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.66:55345)
0810 18:01:39.255648 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.255752 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
I20260810 18:01:39.255937 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.255424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.67:53977)
0810 18:01:39.255712 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.255809 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
W20260810 18:01:39.256266 4009 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:39527 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
W20260810 18:01:39.256342 4147 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:39527 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
I20260810 18:01:39.761792 4220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e68f257ff67e4efebe5bb38e1fafe09b)
I20260810 18:01:39.761929 4221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e68f257ff67e4efebe5bb38e1fafe09b)
I20260810 18:01:39.761933 4220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } }
I20260810 18:01:39.762183 4221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } }
I20260810 18:01:39.762568 4220 leader_election.cc:288] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e68f257ff67e4efebe5bb38e1fafe09b (127.28.83.126:39527), 55c799d9947b47f0a67d47b3053da1ff (127.28.83.125:42411)
I20260810 18:01:39.763355 4221 leader_election.cc:288] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e68f257ff67e4efebe5bb38e1fafe09b (127.28.83.126:39527), 2c754725a56f48c59f4fa09998b76521 (127.28.83.124:37029)
I20260810 18:01:39.763747 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.762767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50486)
0810 18:01:39.763138 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.763250 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":69,"threads_started":1}
W20260810 18:01:39.763942 3664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
I20260810 18:01:39.764115 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.763467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50494)
0810 18:01:39.763918 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.763994 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":59,"threads_started":1}
W20260810 18:01:39.764302 3472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
I20260810 18:01:39.765163 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.764896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50502)
0810 18:01:39.765012 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.765068 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 18:01:39.765311 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.764975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50512)
0810 18:01:39.765101 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.765147 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260810 18:01:39.765301 3664 leader_election.cc:334] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e68f257ff67e4efebe5bb38e1fafe09b (127.28.83.126:39527): Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
W20260810 18:01:39.765480 3472 leader_election.cc:334] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e68f257ff67e4efebe5bb38e1fafe09b (127.28.83.126:39527): Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
I20260810 18:01:39.767307 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.763123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.28.83.125:42411)
0810 18:01:39.763573 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:39.763578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:39.763593 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:01:39.763652 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:39.763654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:39.763704 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:39.764142 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:39.764259 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.765039 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.766372 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.766816 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.766896 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.766943 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:39.767157 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 18:01:39.767212 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":373,"thread_start_us":115,"threads_started":1}
I20260810 18:01:39.767354 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.762787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42411 (local address 127.0.0.1:56510)
0810 18:01:39.763184 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.763202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:39.763340 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:39.763844 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:39.763847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:39.763866 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:39.764007 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.764016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.765173 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.765179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.766261 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.766272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.766934 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.766938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.767039 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.767065 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:01:39.767150 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:01:39.767219 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":59,"threads_started":1}
I20260810 18:01:39.767763 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c754725a56f48c59f4fa09998b76521" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "55c799d9947b47f0a67d47b3053da1ff" is_pre_election: true
I20260810 18:01:39.767926 3495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c754725a56f48c59f4fa09998b76521 in term 1.
I20260810 18:01:39.768241 3663 leader_election.cc:302] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c754725a56f48c59f4fa09998b76521, 55c799d9947b47f0a67d47b3053da1ff; no voters: e68f257ff67e4efebe5bb38e1fafe09b
I20260810 18:01:39.768478 4220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:01:39.768550 4220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e68f257ff67e4efebe5bb38e1fafe09b)
I20260810 18:01:39.768600 4220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:01:39.768688 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.763534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:37029 (local address 127.0.0.1:33778)
0810 18:01:39.763962 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.763977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:39.764136 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:39.765011 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:39.765015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:39.765034 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:39.765133 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.765140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.766228 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.766231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.767322 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.767332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.768241 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.768244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.768332 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.768368 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 18:01:39.768447 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:01:39.768531 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":39,"thread_start_us":69,"threads_started":1}
I20260810 18:01:39.768743 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.763695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.28.83.124:37029)
0810 18:01:39.764518 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:39.764523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:39.764537 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:01:39.764580 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:39.764583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:39.764627 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:39.765488 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:39.765595 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.766113 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.767517 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.768122 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.768248 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.768314 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:39.768483 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 18:01:39.768564 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":78,"threads_started":1}
I20260810 18:01:39.769170 3688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55c799d9947b47f0a67d47b3053da1ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2c754725a56f48c59f4fa09998b76521" is_pre_election: true
I20260810 18:01:39.769965 4220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } }
I20260810 18:01:39.770270 3688 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 55c799d9947b47f0a67d47b3053da1ff in current term 2: Already voted for candidate 2c754725a56f48c59f4fa09998b76521 in this term.
I20260810 18:01:39.770394 4220 leader_election.cc:288] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [CANDIDATE]: Term 2 election: Requested vote from peers e68f257ff67e4efebe5bb38e1fafe09b (127.28.83.126:39527), 55c799d9947b47f0a67d47b3053da1ff (127.28.83.125:42411)
I20260810 18:01:39.770779 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.770512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50524)
0810 18:01:39.770658 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.770717 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 18:01:39.770766 3471 leader_election.cc:302] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55c799d9947b47f0a67d47b3053da1ff; no voters: 2c754725a56f48c59f4fa09998b76521, e68f257ff67e4efebe5bb38e1fafe09b
I20260810 18:01:39.770972 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c754725a56f48c59f4fa09998b76521" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "55c799d9947b47f0a67d47b3053da1ff"
I20260810 18:01:39.771086 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:01:39.771503 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.771268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50540)
0810 18:01:39.771391 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.771447 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:01:39.771617 3664 leader_election.cc:334] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e68f257ff67e4efebe5bb38e1fafe09b (127.28.83.126:39527): Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
I20260810 18:01:39.772286 3495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c754725a56f48c59f4fa09998b76521 in term 2.
I20260810 18:01:39.772401 4221 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:01:39.772585 3663 leader_election.cc:302] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c754725a56f48c59f4fa09998b76521, 55c799d9947b47f0a67d47b3053da1ff; no voters: e68f257ff67e4efebe5bb38e1fafe09b
I20260810 18:01:39.772742 4220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:01:39.772991 4220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [term 2 LEADER]: Becoming Leader. State: Replica: 2c754725a56f48c59f4fa09998b76521, State: Running, Role: LEADER
I20260810 18:01:39.773275 4220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } }
I20260810 18:01:39.774601 4231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c754725a56f48c59f4fa09998b76521. Latest consensus state: current_term: 2 leader_uuid: "2c754725a56f48c59f4fa09998b76521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:39.774696 4231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:39.774972 4233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:39.777518 4233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:39.777992 4233 catalog_manager.cc:1259] Loaded cluster ID: bcfc9d0c64304b67b1ea56372f488b35
I20260810 18:01:39.778038 4233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:39.778754 4233 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:39.779173 4233 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Loaded TSK: 0
I20260810 18:01:39.779860 4233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:01:39.815790 3676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:39.819136 3676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:39.822165 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.821877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50556)
0810 18:01:39.821997 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.822081 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 18:01:39.822278 3495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [term 2 FOLLOWER]: Refusing update from remote peer 2c754725a56f48c59f4fa09998b76521: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 18:01:39.822796 4220 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 18:01:39.822904 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.822656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50568)
0810 18:01:39.822788 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.822846 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 18:01:39.823017 3664 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 -> Peer e68f257ff67e4efebe5bb38e1fafe09b (127.28.83.126:39527): Couldn't send request to peer e68f257ff67e4efebe5bb38e1fafe09b. Status: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:01:39.824432 4221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c754725a56f48c59f4fa09998b76521. Latest consensus state: current_term: 2 leader_uuid: "2c754725a56f48c59f4fa09998b76521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:39.824571 4221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:01:39.825948 4231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2c754725a56f48c59f4fa09998b76521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:39.826089 4231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:39.827241 4221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2c754725a56f48c59f4fa09998b76521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68f257ff67e4efebe5bb38e1fafe09b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 39527 } } peers { permanent_uuid: "55c799d9947b47f0a67d47b3053da1ff" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2c754725a56f48c59f4fa09998b76521" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 37029 } attrs { promote: false } } }
I20260810 18:01:39.827376 4221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c799d9947b47f0a67d47b3053da1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:01:39.836759 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.831937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:35521 (local address 127.0.0.1:43294)
0810 18:01:39.832370 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.832386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:39.832466 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:39.832879 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:39.832881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:39.832895 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:39.833024 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.833031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.834187 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.834190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.835376 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.835387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.836296 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.836299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.836388 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.836426 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 18:01:39.836498 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:01:39.836582 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359}
I20260810 18:01:39.836961 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.832444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:44715 (local address 127.0.0.1:38872)
0810 18:01:39.832707 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.832720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:39.832805 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:39.833256 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:39.833259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:39.833280 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:39.833395 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.833401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.834481 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.834485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.835616 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.835626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.836506 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.836509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.836599 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.836622 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:01:39.836715 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 18:01:39.836791 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":103,"spinlock_wait_cycles":384}
I20260810 18:01:39.836999 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.832723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.28.83.66:44715)
0810 18:01:39.832979 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:39.832983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:39.832996 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:01:39.833052 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:39.833057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:39.833104 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:39.833525 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:39.833714 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.834334 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.835756 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.836374 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.836498 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.836561 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:39.836749 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 18:01:39.836847 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260810 18:01:39.837287 4240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.832118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.28.83.67:35521)
0810 18:01:39.832605 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:39.832611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:39.832628 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:01:39.832686 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:39.832689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:39.832749 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:39.833141 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:39.833292 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.834067 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.835528 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.836171 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.836279 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.836347 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:39.837051 (+ 704us) server_negotiation.cc:299] Negotiation successful
0810 18:01:39.837155 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":102,"threads_started":1}
I20260810 18:01:39.837678 3938 tablet_service.cc:1467] Processing CreateTablet for tablet 9d57228387b24236bed17ccbbe7cd2de (DEFAULT_TABLE table=test-table [id=47f9f0b2be454129a698ce1f48540ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:39.837829 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.832448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42123 (local address 127.0.0.1:54428)
0810 18:01:39.833122 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.833139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:39.833272 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:39.833604 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:39.833607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:39.833623 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:39.833721 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.833727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.834661 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.834664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.835868 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:39.835877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.837431 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.837435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:39.837527 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.837552 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:01:39.837636 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 18:01:39.837725 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":456,"thread_start_us":108,"threads_started":1}
I20260810 18:01:39.837855 4080 tablet_service.cc:1467] Processing CreateTablet for tablet 9d57228387b24236bed17ccbbe7cd2de (DEFAULT_TABLE table=test-table [id=47f9f0b2be454129a698ce1f48540ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:39.838006 3938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d57228387b24236bed17ccbbe7cd2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:39.838097 4080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d57228387b24236bed17ccbbe7cd2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:39.840776 4245 tablet_bootstrap.cc:492] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: Bootstrap starting.
I20260810 18:01:39.841583 4241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.832656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.28.83.65:42123)
0810 18:01:39.832933 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:39.832938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:39.833213 (+ 275us) server_negotiation.cc:407] Connection header received
0810 18:01:39.833434 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:39.833438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:39.833490 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:39.833582 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:39.833838 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.834544 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.836014 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:39.836645 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:39.836727 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:39.836793 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:39.841280 (+ 4487us) server_negotiation.cc:299] Negotiation successful
0810 18:01:39.841402 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
I20260810 18:01:39.841918 4245 tablet_bootstrap.cc:654] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:39.842198 3797 tablet_service.cc:1467] Processing CreateTablet for tablet 9d57228387b24236bed17ccbbe7cd2de (DEFAULT_TABLE table=test-table [id=47f9f0b2be454129a698ce1f48540ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:39.842486 3797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d57228387b24236bed17ccbbe7cd2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:39.842577 4246 tablet_bootstrap.cc:492] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: Bootstrap starting.
I20260810 18:01:39.843247 4245 tablet_bootstrap.cc:492] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: No bootstrap required, opened a new log
I20260810 18:01:39.843327 4245 ts_tablet_manager.cc:1397] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:39.843377 4246 tablet_bootstrap.cc:654] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:39.843743 4245 raft_consensus.cc:348] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } }
I20260810 18:01:39.843858 4245 raft_consensus.cc:374] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:39.843892 4245 raft_consensus.cc:729] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa7b864070f42dbaf8af4a9ba72a0a8, State: Initialized, Role: FOLLOWER
I20260810 18:01:39.844033 4245 consensus_queue.cc:260] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } }
I20260810 18:01:39.844395 4245 ts_tablet_manager.cc:1428] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:39.844533 4246 tablet_bootstrap.cc:492] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: No bootstrap required, opened a new log
I20260810 18:01:39.844612 4246 ts_tablet_manager.cc:1397] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:39.846581 4246 raft_consensus.cc:348] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } }
I20260810 18:01:39.846725 4246 raft_consensus.cc:374] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:39.846769 4246 raft_consensus.cc:729] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48a1c3a95094db1a8be66645ecfccb7, State: Initialized, Role: FOLLOWER
I20260810 18:01:39.846992 4008 heartbeater.cc:503] Master 127.28.83.124:37029 was elected leader, sending a full tablet report...
I20260810 18:01:39.847323 4246 consensus_queue.cc:260] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } }
I20260810 18:01:39.847725 4246 ts_tablet_manager.cc:1428] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:39.848191 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.844849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.66:60129)
0810 18:01:39.848005 (+ 3156us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.848076 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3071,"thread_start_us":99,"threads_started":1}
I20260810 18:01:39.848469 4250 tablet_bootstrap.cc:492] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce: Bootstrap starting.
I20260810 18:01:39.849233 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.848656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.66:60327)
0810 18:01:39.849118 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.849173 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393}
I20260810 18:01:39.849365 4250 tablet_bootstrap.cc:654] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:39.850078 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.848214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.67:53287)
0810 18:01:39.849642 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.849699 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1345,"thread_start_us":103,"threads_started":1}
I20260810 18:01:39.850530 4146 heartbeater.cc:503] Master 127.28.83.124:37029 was elected leader, sending a full tablet report...
I20260810 18:01:39.851025 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.850524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.67:45033)
0810 18:01:39.850911 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.850959 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":303}
I20260810 18:01:39.852087 4250 tablet_bootstrap.cc:492] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce: No bootstrap required, opened a new log
I20260810 18:01:39.852166 4250 ts_tablet_manager.cc:1397] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260810 18:01:39.852505 4250 raft_consensus.cc:348] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } }
I20260810 18:01:39.852593 4250 raft_consensus.cc:374] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:39.852614 4250 raft_consensus.cc:729] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f939c8313248218f4d26b864ae4dce, State: Initialized, Role: FOLLOWER
I20260810 18:01:39.852769 4250 consensus_queue.cc:260] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } }
I20260810 18:01:39.853132 4250 ts_tablet_manager.cc:1428] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:01:39.854017 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.853605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.65:44105)
0810 18:01:39.853852 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.853911 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":110,"threads_started":1}
I20260810 18:01:39.854596 3865 heartbeater.cc:503] Master 127.28.83.124:37029 was elected leader, sending a full tablet report...
I20260810 18:01:39.855041 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:39.854453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.65:33545)
0810 18:01:39.854912 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:39.854976 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
W20260810 18:01:39.857187 4011 tablet.cc:2369] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:01:39.898082 3868 tablet.cc:2369] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:01:40.066548 4150 tablet.cc:2369] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:40.135293 4255 raft_consensus.cc:482] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:40.135452 4255 raft_consensus.cc:504] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } }
I20260810 18:01:40.136117 4255 leader_election.cc:288] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b48a1c3a95094db1a8be66645ecfccb7 (127.28.83.67:35521), 7fa7b864070f42dbaf8af4a9ba72a0a8 (127.28.83.66:44715)
I20260810 18:01:40.136653 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d57228387b24236bed17ccbbe7cd2de" candidate_uuid: "63f939c8313248218f4d26b864ae4dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48a1c3a95094db1a8be66645ecfccb7" is_pre_election: true
I20260810 18:01:40.136686 3961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d57228387b24236bed17ccbbe7cd2de" candidate_uuid: "63f939c8313248218f4d26b864ae4dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" is_pre_election: true
I20260810 18:01:40.136801 4100 raft_consensus.cc:2454] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f939c8313248218f4d26b864ae4dce in term 0.
I20260810 18:01:40.136901 3961 raft_consensus.cc:2454] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f939c8313248218f4d26b864ae4dce in term 0.
I20260810 18:01:40.137218 3750 leader_election.cc:302] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f939c8313248218f4d26b864ae4dce, b48a1c3a95094db1a8be66645ecfccb7; no voters:
I20260810 18:01:40.137428 4255 raft_consensus.cc:2781] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:40.137494 4255 raft_consensus.cc:482] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:40.137539 4255 raft_consensus.cc:3037] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:40.138681 4255 raft_consensus.cc:504] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } }
I20260810 18:01:40.139052 4255 leader_election.cc:288] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [CANDIDATE]: Term 1 election: Requested vote from peers b48a1c3a95094db1a8be66645ecfccb7 (127.28.83.67:35521), 7fa7b864070f42dbaf8af4a9ba72a0a8 (127.28.83.66:44715)
I20260810 18:01:40.139437 3961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d57228387b24236bed17ccbbe7cd2de" candidate_uuid: "63f939c8313248218f4d26b864ae4dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8"
I20260810 18:01:40.139444 4100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d57228387b24236bed17ccbbe7cd2de" candidate_uuid: "63f939c8313248218f4d26b864ae4dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48a1c3a95094db1a8be66645ecfccb7"
I20260810 18:01:40.139590 3961 raft_consensus.cc:3037] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:40.139611 4100 raft_consensus.cc:3037] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:40.140514 3961 raft_consensus.cc:2454] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f939c8313248218f4d26b864ae4dce in term 1.
I20260810 18:01:40.140583 4100 raft_consensus.cc:2454] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f939c8313248218f4d26b864ae4dce in term 1.
I20260810 18:01:40.140780 3750 leader_election.cc:302] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f939c8313248218f4d26b864ae4dce, 7fa7b864070f42dbaf8af4a9ba72a0a8; no voters:
I20260810 18:01:40.141019 4255 raft_consensus.cc:2781] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:40.141278 4255 raft_consensus.cc:686] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 1 LEADER]: Becoming Leader. State: Replica: 63f939c8313248218f4d26b864ae4dce, State: Running, Role: LEADER
I20260810 18:01:40.141482 4255 consensus_queue.cc:237] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } }
I20260810 18:01:40.142037 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.141802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.65:55857)
0810 18:01:40.141910 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:40.141972 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 18:01:40.143344 3677 catalog_manager.cc:5495] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce reported cstate change: term changed from 0 to 1, leader changed from <none> to 63f939c8313248218f4d26b864ae4dce (127.28.83.65). New cstate: current_term: 1 leader_uuid: "63f939c8313248218f4d26b864ae4dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63f939c8313248218f4d26b864ae4dce" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 } health_report { overall_health: UNKNOWN } } }
I20260810 18:01:40.203133 4241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.200249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.28.83.65:42123)
0810 18:01:40.200410 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:40.200416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:40.200438 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:01:40.200488 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:40.200490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:40.200535 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:40.200615 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:40.200856 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:40.201381 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:40.202245 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:40.202455 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:40.202533 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:40.202765 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:40.202834 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:40.202883 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:40.202967 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 18:01:40.203013 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 18:01:40.205528 4100 raft_consensus.cc:1261] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [term 1 FOLLOWER]: Refusing update from remote peer 63f939c8313248218f4d26b864ae4dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:40.205775 3961 raft_consensus.cc:1261] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 1 FOLLOWER]: Refusing update from remote peer 63f939c8313248218f4d26b864ae4dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:40.206045 4255 consensus_queue.cc:1035] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b48a1c3a95094db1a8be66645ecfccb7" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:40.206238 4257 consensus_queue.cc:1035] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa7b864070f42dbaf8af4a9ba72a0a8" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:40.208199 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.207894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.66:48267)
0810 18:01:40.208048 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:40.208119 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 18:01:40.209317 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.208260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.67:46913)
0810 18:01:40.208397 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:40.209241 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 18:01:40.211956 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.211714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.65:39963)
0810 18:01:40.211847 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:40.211898 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 18:01:40.211956 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.211682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.67:57815)
0810 18:01:40.211838 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:40.211889 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 18:01:40.212734 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.212465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.28.83.66:60663)
0810 18:01:40.212608 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:40.212663 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 18:01:40.217542 3676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:40.217765 3676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:40.220237 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.219971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50576)
0810 18:01:40.220096 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:40.220153 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 18:01:40.221122 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.220899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:39527 (local address 127.0.0.1:50582)
0810 18:01:40.221003 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:40.221059 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 18:01:40.221479 3676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c754725a56f48c59f4fa09998b76521: Sending DeleteTablet for 3 replicas of tablet 9d57228387b24236bed17ccbbe7cd2de
I20260810 18:01:40.222153 4080 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d57228387b24236bed17ccbbe7cd2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:40 UTC) from {username='oryx', principal='oryx/127.28.83.124@KRBTEST.COM'} at 127.0.0.1:43294
I20260810 18:01:40.222536 4269 tablet_replica.cc:330] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: stopping tablet replica
I20260810 18:01:40.222524 3797 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d57228387b24236bed17ccbbe7cd2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:40 UTC) from {username='oryx', principal='oryx/127.28.83.124@KRBTEST.COM'} at 127.0.0.1:54428
I20260810 18:01:40.222671 4269 raft_consensus.cc:2229] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:40.222824 4269 raft_consensus.cc:2258] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:40.222889 4270 tablet_replica.cc:330] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce: stopping tablet replica
I20260810 18:01:40.223177 4270 raft_consensus.cc:2229] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:40.223413 4270 raft_consensus.cc:2258] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:40.223511 3938 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d57228387b24236bed17ccbbe7cd2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:40 UTC) from {username='oryx', principal='oryx/127.28.83.124@KRBTEST.COM'} at 127.0.0.1:38872
I20260810 18:01:40.223740 4269 ts_tablet_manager.cc:1905] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:40.224005 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 3636
I20260810 18:01:40.224098 4270 ts_tablet_manager.cc:1905] T 9d57228387b24236bed17ccbbe7cd2de P 63f939c8313248218f4d26b864ae4dce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:40.226846 4269 ts_tablet_manager.cc:1918] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:40.226931 4269 log.cc:1196] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/9d57228387b24236bed17ccbbe7cd2de
I20260810 18:01:40.227211 4269 ts_tablet_manager.cc:1939] T 9d57228387b24236bed17ccbbe7cd2de P b48a1c3a95094db1a8be66645ecfccb7: Deleting consensus metadata
I20260810 18:01:40.227325 4271 tablet_replica.cc:330] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: stopping tablet replica
I20260810 18:01:40.227451 4271 raft_consensus.cc:2229] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:40.227592 4271 raft_consensus.cc:2258] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:40.228336 4271 ts_tablet_manager.cc:1905] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:40.229758 3663 catalog_manager.cc:4841] TS b48a1c3a95094db1a8be66645ecfccb7 (127.28.83.67:35521): tablet 9d57228387b24236bed17ccbbe7cd2de (table test-table [id=47f9f0b2be454129a698ce1f48540ca9]) successfully deleted
I20260810 18:01:40.230481 4271 ts_tablet_manager.cc:1918] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:40.230548 4271 log.cc:1196] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/9d57228387b24236bed17ccbbe7cd2de
I20260810 18:01:40.230757 4271 ts_tablet_manager.cc:1939] T 9d57228387b24236bed17ccbbe7cd2de P 7fa7b864070f42dbaf8af4a9ba72a0a8: Deleting consensus metadata
I20260810 18:01:40.231351 3663 catalog_manager.cc:4841] TS 7fa7b864070f42dbaf8af4a9ba72a0a8 (127.28.83.66:44715): tablet 9d57228387b24236bed17ccbbe7cd2de (table test-table [id=47f9f0b2be454129a698ce1f48540ca9]) successfully deleted
I20260810 18:01:40.234045 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:40.233795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42123 (local address 127.0.0.1:54444)
0810 18:01:40.233924 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:40.233978 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 18:01:40.234297 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 3873
W20260810 18:01:40.234265 3662 catalog_manager.cc:4584] TS 63f939c8313248218f4d26b864ae4dce (127.28.83.65:42123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d57228387b24236bed17ccbbe7cd2de: Network error: Client connection negotiation failed: client connection to 127.28.83.65:42123: connect: Connection refused (error 111)
I20260810 18:01:40.243449 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 4015
I20260810 18:01:40.250793 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 3343
I20260810 18:01:40.258234 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 3449
2026-08-10T18:01:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5705 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:40 dist-test-slave-srgv krb5kdc[4274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:40 dist-test-slave-srgv krb5kdc[4274](info): set up 2 sockets
Aug 10 18:01:40 dist-test-slave-srgv krb5kdc[4274](info): commencing operation
krb5kdc: starting...
W20260810 18:01:41.317015 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:41 dist-test-slave-srgv krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.126.keytab.
Entry for principal oryx/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.126.keytab.
I20260810 18:01:41.444089 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37985
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.126.keytab
--principal=oryx/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:41.556370 4289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:41.556692 4289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:41.556737 4289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:41.560923 4289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:41.560990 4289 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:01:41.561008 4289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:41.561026 4289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:41.561043 4289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:41.561100 4289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:41.566679 4289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:37985
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.126.keytab
--principal=oryx/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:41.567816 4289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:41.568912 4289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:41.577334 4295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:41.577442 4294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:41.577334 4297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:41.577757 4289 server_base.cc:1034] running on GCE node
I20260810 18:01:41.578306 4289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:41.579421 4289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:41.580585 4289 hybrid_clock.cc:648] HybridClock initialized: now 1786384901580556 us; error 43 us; skew 500 ppm
Aug 10 18:01:41 dist-test-slave-srgv krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384901, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:41.583423 4289 init.cc:377] Logged in from keytab as oryx/127.28.83.126@KRBTEST.COM (short username oryx)
I20260810 18:01:41.584378 4289 webserver.cc:460] Webserver started at http://127.28.83.126:39645/ using document root <none> and password file <none>
I20260810 18:01:41.584940 4289 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:41.585003 4289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:41.585253 4289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:41.587425 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "acbf32bf1543412e9c085d41df9be5bf"
format_stamp: "Formatted at 2026-08-10 18:01:41 on dist-test-slave-srgv"
server_key: "3096ce6b6d97a818b89aea1f97551cad"
server_key_iv: "e789fdd1fce02c6fb01cfe056f90492f"
server_key_version: "encryptionkey@0"
I20260810 18:01:41.588027 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "acbf32bf1543412e9c085d41df9be5bf"
format_stamp: "Formatted at 2026-08-10 18:01:41 on dist-test-slave-srgv"
server_key: "3096ce6b6d97a818b89aea1f97551cad"
server_key_iv: "e789fdd1fce02c6fb01cfe056f90492f"
server_key_version: "encryptionkey@0"
I20260810 18:01:41.591998 4289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 18:01:41.595196 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:41.596609 4289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:01:41.596748 4289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "acbf32bf1543412e9c085d41df9be5bf"
format_stamp: "Formatted at 2026-08-10 18:01:41 on dist-test-slave-srgv"
server_key: "3096ce6b6d97a818b89aea1f97551cad"
server_key_iv: "e789fdd1fce02c6fb01cfe056f90492f"
server_key_version: "encryptionkey@0"
I20260810 18:01:41.596877 4289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:41.641736 4289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:41.645269 4289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:41.645514 4289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:41.654072 4289 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:37985
I20260810 18:01:41.654111 4367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:37985 every 8 connection(s)
I20260810 18:01:41.655073 4289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:41.658000 4368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:41.661227 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 4289
I20260810 18:01:41.661381 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:41.661686 29005 external_mini_cluster.cc:1428] Setting key 1abce44147bd823292b0c035bd7f3687
I20260810 18:01:41.664273 4368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf: Bootstrap starting.
I20260810 18:01:41.666953 4368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:41.667881 4368 log.cc:824] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:41.669922 4368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf: No bootstrap required, opened a new log
Aug 10 18:01:41 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:41.673188 4368 raft_consensus.cc:348] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbf32bf1543412e9c085d41df9be5bf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:01:41.673481 4368 raft_consensus.cc:374] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:41.673563 4368 raft_consensus.cc:729] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acbf32bf1543412e9c085d41df9be5bf, State: Initialized, Role: FOLLOWER
I20260810 18:01:41.674094 4368 consensus_queue.cc:260] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbf32bf1543412e9c085d41df9be5bf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:01:41.674271 4368 raft_consensus.cc:388] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:41.674345 4368 raft_consensus.cc:482] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:41.674440 4368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:41.675705 4368 raft_consensus.cc:504] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbf32bf1543412e9c085d41df9be5bf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:01:41.676066 4368 leader_election.cc:302] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acbf32bf1543412e9c085d41df9be5bf; no voters:
I20260810 18:01:41.676440 4368 leader_election.cc:288] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:41.676489 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:41.663055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.28.83.126:37985)
0810 18:01:41.663484 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:41.663501 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:41.663535 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:01:41.664164 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:41.664181 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:41.664557 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:41.664846 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:41.665852 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.666847 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.668061 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.668515 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.668701 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:41.671356 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:41.671361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:41.671374 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:41.671400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:41.673143 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:41.673685 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:41.673690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:41.673696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:41.673841 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:41.674127 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:41.674130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:41.674132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:41.674688 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:41.674899 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:41.675138 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 18:01:41.675458 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":142,"threads_started":1}
I20260810 18:01:41.676558 4373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:41.676811 4373 raft_consensus.cc:686] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [term 1 LEADER]: Becoming Leader. State: Replica: acbf32bf1543412e9c085d41df9be5bf, State: Running, Role: LEADER
I20260810 18:01:41.677098 4373 consensus_queue.cc:237] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbf32bf1543412e9c085d41df9be5bf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:01:41.677752 4368 sys_catalog.cc:564] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:41.678316 4375 sys_catalog.cc:455] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acbf32bf1543412e9c085d41df9be5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbf32bf1543412e9c085d41df9be5bf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } } }
I20260810 18:01:41.678432 4375 sys_catalog.cc:458] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:41.678639 4375 sys_catalog.cc:455] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader acbf32bf1543412e9c085d41df9be5bf. Latest consensus state: current_term: 1 leader_uuid: "acbf32bf1543412e9c085d41df9be5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbf32bf1543412e9c085d41df9be5bf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } } }
I20260810 18:01:41.678721 4375 sys_catalog.cc:458] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:41.679116 4379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:41.681869 4379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:41.687518 4379 catalog_manager.cc:1347] Generated new cluster ID: 434440a63c4041618dc08306d727490f
I20260810 18:01:41.687592 4379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:41.692915 4379 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:41.693890 4379 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:41.701788 4379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf: Generated new TSK 0
I20260810 18:01:41.702535 4379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab.
Entry for principal oryx/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab.
I20260810 18:01:41.737895 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:37985
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab
--principal=oryx/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:41.851567 4394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:41.851907 4394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:41.851989 4394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:41.856019 4394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:41.856120 4394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:41.856197 4394 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:01:41.856278 4394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:41.861538 4394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab
--principal=oryx/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:41.862666 4394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:41.863703 4394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:41.873252 4399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:41.873350 4394 server_base.cc:1034] running on GCE node
W20260810 18:01:41.873315 4400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:41.873252 4402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:41.873958 4394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:41.874665 4394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:41.875861 4394 hybrid_clock.cc:648] HybridClock initialized: now 1786384901875842 us; error 31 us; skew 500 ppm
Aug 10 18:01:41 dist-test-slave-srgv krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384901, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:41.878780 4394 init.cc:377] Logged in from keytab as oryx/127.28.83.65@KRBTEST.COM (short username oryx)
I20260810 18:01:41.879796 4394 webserver.cc:460] Webserver started at http://127.28.83.65:37215/ using document root <none> and password file <none>
I20260810 18:01:41.880355 4394 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:41.880446 4394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:41.880681 4394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:41.882792 4394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "362c009765cf463fb2d60af0f9556f30"
format_stamp: "Formatted at 2026-08-10 18:01:41 on dist-test-slave-srgv"
server_key: "0e1a27d0a0ca0aba2627e8e9fa043e0b"
server_key_iv: "b3cc88536a89c8f28e602afa0fc78d1f"
server_key_version: "encryptionkey@0"
I20260810 18:01:41.883376 4394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "362c009765cf463fb2d60af0f9556f30"
format_stamp: "Formatted at 2026-08-10 18:01:41 on dist-test-slave-srgv"
server_key: "0e1a27d0a0ca0aba2627e8e9fa043e0b"
server_key_iv: "b3cc88536a89c8f28e602afa0fc78d1f"
server_key_version: "encryptionkey@0"
I20260810 18:01:41.887315 4394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:01:41.889784 4409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:41.891036 4394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:41.891193 4394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "362c009765cf463fb2d60af0f9556f30"
format_stamp: "Formatted at 2026-08-10 18:01:41 on dist-test-slave-srgv"
server_key: "0e1a27d0a0ca0aba2627e8e9fa043e0b"
server_key_iv: "b3cc88536a89c8f28e602afa0fc78d1f"
server_key_version: "encryptionkey@0"
I20260810 18:01:41.891315 4394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:41.916394 4394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:41.919353 4394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:41.919579 4394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:41.921051 4394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:41.921139 4394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:41.921217 4394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:41.921260 4394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:01:41 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384901, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:41.932363 4394 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:35143
I20260810 18:01:41.932523 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:35143 every 8 connection(s)
I20260810 18:01:41.933590 4394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:41.935453 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 4394
I20260810 18:01:41.935592 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:41.935940 29005 external_mini_cluster.cc:1428] Setting key 24300dfa8ae020900c0dc2c3d02e1421
I20260810 18:01:41.936273 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:41.922045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:42959 (local address 127.28.83.126:37985)
0810 18:01:41.922221 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:41.922229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:41.923074 (+ 845us) server_negotiation.cc:407] Connection header received
0810 18:01:41.924238 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:41.924242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:41.924310 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:41.924417 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:41.925837 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.926729 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.928642 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.929039 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.929165 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:41.932445 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:41.932451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:41.932453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:41.932486 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:41.934215 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:41.934768 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:41.934771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:41.934772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:41.934853 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:41.935104 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:41.935105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:41.935107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:41.935278 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:41.935350 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:41.935987 (+ 637us) server_negotiation.cc:299] Negotiation successful
0810 18:01:41.936143 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:01:41.937172 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:41.922441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:42959)
0810 18:01:41.922901 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:41.922950 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:41.923973 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:41.924616 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:41.924623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:41.924988 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:41.925678 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:41.925705 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.926884 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.926889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:41.928489 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:41.928502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.929231 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.929236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:41.929492 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:41.930158 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:41.930181 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:41.932243 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:41.934362 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:41.934370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:41.934384 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:41.934670 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:41.934929 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:41.934931 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:41.934932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:41.935045 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:41.935384 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:41.935391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:41.935721 (+ 330us) client_negotiation.cc:769] Sending connection context
0810 18:01:41.935939 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 18:01:41.936282 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":142,"threads_started":1}
I20260810 18:01:41.938563 4526 heartbeater.cc:347] Connected to a master server at 127.28.83.126:37985
I20260810 18:01:41.938714 4526 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:41.939215 4526 heartbeater.cc:511] Master 127.28.83.126:37985 requested a full tablet report, sending...
I20260810 18:01:41.940644 4320 ts_manager.cc:194] Registered new tserver with Master: 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143)
I20260810 18:01:41.942233 4320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:42959
WARNING: no policy specified for oryx/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:01:41.949833 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:41.941044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:41199)
0810 18:01:41.941357 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:41.941373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:41.941500 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:41.941782 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:41.941785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:41.941994 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:41.942124 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:41.942132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.943214 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.943218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:41.944013 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:41.944022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.944719 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.944726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:41.944846 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:41.945563 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:41.945583 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:41.946117 (+ 534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:41.948234 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:41.948237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:41.948240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:41.948482 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:41.948788 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:41.948791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:41.948792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:41.948876 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:41.949278 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:41.949281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:41.949370 (+ 89us) client_negotiation.cc:769] Sending connection context
0810 18:01:41.949492 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 18:01:41.949629 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20260810 18:01:41.950357 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:41.941177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:41199 (local address 127.28.83.126:37985)
0810 18:01:41.941311 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:41.941315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:41.941467 (+ 152us) server_negotiation.cc:407] Connection header received
0810 18:01:41.941598 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:41.941603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:41.941662 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:41.941749 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:41.942260 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.943096 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.944183 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:41.944576 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:41.944703 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:41.946291 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:41.946297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:41.946301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:41.946334 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:41.948108 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:41.948592 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:41.948595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:41.948597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:41.948678 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:41.948969 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:41.948972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:41.948973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:41.949156 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:41.949236 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:41.950088 (+ 852us) server_negotiation.cc:299] Negotiation successful
0810 18:01:41.950197 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab.
Entry for principal oryx/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab.
I20260810 18:01:41.971976 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:37985
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab
--principal=oryx/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:42.082744 4535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:42.083007 4535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:42.083052 4535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:42.087113 4535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:42.087177 4535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:42.087220 4535 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:01:42.087255 4535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:42.092658 4535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab
--principal=oryx/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:42.093801 4535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:42.094846 4535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:42.104590 4540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:42.104542 4541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:42.104548 4543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:42.105181 4535 server_base.cc:1034] running on GCE node
I20260810 18:01:42.105643 4535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:42.106220 4535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:42.107372 4535 hybrid_clock.cc:648] HybridClock initialized: now 1786384902107333 us; error 47 us; skew 500 ppm
Aug 10 18:01:42 dist-test-slave-srgv krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384902, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:42.110216 4535 init.cc:377] Logged in from keytab as oryx/127.28.83.66@KRBTEST.COM (short username oryx)
I20260810 18:01:42.111032 4535 webserver.cc:460] Webserver started at http://127.28.83.66:46209/ using document root <none> and password file <none>
I20260810 18:01:42.111568 4535 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:42.111660 4535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:42.111892 4535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:42.113984 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "7ac3e7a2044f40d88038add091b46dcf"
format_stamp: "Formatted at 2026-08-10 18:01:42 on dist-test-slave-srgv"
server_key: "89b2dcc58c6f3fd0698d7abee1a7acff"
server_key_iv: "dc43e23ca88c692c70acfefdc4355ed2"
server_key_version: "encryptionkey@0"
I20260810 18:01:42.114563 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "7ac3e7a2044f40d88038add091b46dcf"
format_stamp: "Formatted at 2026-08-10 18:01:42 on dist-test-slave-srgv"
server_key: "89b2dcc58c6f3fd0698d7abee1a7acff"
server_key_iv: "dc43e23ca88c692c70acfefdc4355ed2"
server_key_version: "encryptionkey@0"
I20260810 18:01:42.118316 4535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:01:42.120704 4550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:42.121960 4535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:42.122087 4535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "7ac3e7a2044f40d88038add091b46dcf"
format_stamp: "Formatted at 2026-08-10 18:01:42 on dist-test-slave-srgv"
server_key: "89b2dcc58c6f3fd0698d7abee1a7acff"
server_key_iv: "dc43e23ca88c692c70acfefdc4355ed2"
server_key_version: "encryptionkey@0"
I20260810 18:01:42.122201 4535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:42.140687 4535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:42.144127 4535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:42.144352 4535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:42.145936 4535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:42.146008 4535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:42.146076 4535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:42.146119 4535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:01:42 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384902, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:42.157750 4535 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:42001
I20260810 18:01:42.157871 4665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:42001 every 8 connection(s)
I20260810 18:01:42.158658 4535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:42.159008 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 4535
I20260810 18:01:42.159174 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:42.159492 29005 external_mini_cluster.cc:1428] Setting key a398f6efa64515fa43a75094cb8d86d5
I20260810 18:01:42.160804 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.146656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:35601 (local address 127.28.83.126:37985)
0810 18:01:42.146842 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.146847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.148318 (+ 1471us) server_negotiation.cc:407] Connection header received
0810 18:01:42.149352 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.149355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.149418 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.149506 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:42.151197 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.151913 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.153068 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.153319 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.153480 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.156679 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:42.156683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:42.156686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:42.156713 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:42.158366 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.158993 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.159000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.159004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.159112 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.159665 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.159668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.159670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.159903 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:42.160041 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.160513 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.160632 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 18:01:42.161670 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.147119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:35601)
0810 18:01:42.148171 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.148212 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.149080 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.149965 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.149989 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.150465 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:42.151027 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.151049 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.152047 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.152051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.152936 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.152945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.153518 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.153522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.153773 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.154497 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:42.154521 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:42.156508 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:42.158486 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.158492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.158505 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.158852 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.159311 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.159318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.159321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.159504 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.160038 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:42.160044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:42.160282 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.160466 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.160768 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":147,"threads_started":1}
I20260810 18:01:42.163102 4667 heartbeater.cc:347] Connected to a master server at 127.28.83.126:37985
I20260810 18:01:42.163242 4667 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:42.163806 4667 heartbeater.cc:511] Master 127.28.83.126:37985 requested a full tablet report, sending...
I20260810 18:01:42.164893 4321 ts_manager.cc:194] Registered new tserver with Master: 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001)
I20260810 18:01:42.165700 4321 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:35601
WARNING: no policy specified for oryx/127.28.83.67@KRBTEST.COM; defaulting to no policy
I20260810 18:01:42.172508 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.165326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:41773 (local address 127.28.83.126:37985)
0810 18:01:42.165478 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.165482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.165804 (+ 322us) server_negotiation.cc:407] Connection header received
0810 18:01:42.165985 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.165988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.166045 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.166140 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:42.166644 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.167134 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.167953 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.168171 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.168282 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.169375 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:42.169379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:42.169382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:42.169437 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:42.170803 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.171237 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.171240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.171242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.171319 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.171616 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.171620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.171622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.171828 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:42.171938 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.172268 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.172376 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:01:42.172514 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.165236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:41773)
0810 18:01:42.165684 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.165698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.165877 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.166171 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.166174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.166388 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:42.166525 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.166530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.167262 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.167265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.167832 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.167839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.168287 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.168290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.168380 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.168901 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:42.168915 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:42.169241 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:42.170932 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.170935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.170937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.171131 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.171411 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.171413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.171415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.171525 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.171929 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:42.171932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:42.172030 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.172141 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.172295 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":21,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab.
Entry for principal oryx/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab.
I20260810 18:01:42.192523 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:37985
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab
--principal=oryx/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:42.306665 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:42.306978 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:42.307056 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:42.311188 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:42.311296 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:42.311370 4676 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:01:42.311453 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:42.316844 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab
--principal=oryx/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:42.318027 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:42.319041 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:42.328697 4682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:42.328697 4681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:42.328720 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:42.329011 4676 server_base.cc:1034] running on GCE node
I20260810 18:01:42.329468 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:42.330075 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:42.331243 4676 hybrid_clock.cc:648] HybridClock initialized: now 1786384902331228 us; error 28 us; skew 500 ppm
Aug 10 18:01:42 dist-test-slave-srgv krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384902, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:42.334097 4676 init.cc:377] Logged in from keytab as oryx/127.28.83.67@KRBTEST.COM (short username oryx)
I20260810 18:01:42.334955 4676 webserver.cc:460] Webserver started at http://127.28.83.67:36011/ using document root <none> and password file <none>
I20260810 18:01:42.335525 4676 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:42.335615 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:42.335873 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:42.338011 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1"
format_stamp: "Formatted at 2026-08-10 18:01:42 on dist-test-slave-srgv"
server_key: "c0e848dc1d4ce5b0b3ab6eeb218999c1"
server_key_iv: "364728866de90b13966e0bfb1eb85a83"
server_key_version: "encryptionkey@0"
I20260810 18:01:42.338610 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1"
format_stamp: "Formatted at 2026-08-10 18:01:42 on dist-test-slave-srgv"
server_key: "c0e848dc1d4ce5b0b3ab6eeb218999c1"
server_key_iv: "364728866de90b13966e0bfb1eb85a83"
server_key_version: "encryptionkey@0"
I20260810 18:01:42.342504 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:01:42.344830 4691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:42.346091 4676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:42.346215 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1"
format_stamp: "Formatted at 2026-08-10 18:01:42 on dist-test-slave-srgv"
server_key: "c0e848dc1d4ce5b0b3ab6eeb218999c1"
server_key_iv: "364728866de90b13966e0bfb1eb85a83"
server_key_version: "encryptionkey@0"
I20260810 18:01:42.346324 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:42.371178 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:42.374351 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:42.374578 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:42.376067 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:42.376147 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:42.376214 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:42.376266 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:01:42 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384902, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:42.387614 4676 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:34485
I20260810 18:01:42.388048 4807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:34485 every 8 connection(s)
I20260810 18:01:42.388682 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:42.389972 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 4676
I20260810 18:01:42.390105 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:42.390391 29005 external_mini_cluster.cc:1428] Setting key eac262f63766cf9a998144c10ba3b3eb
I20260810 18:01:42.391175 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.376866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:45275 (local address 127.28.83.126:37985)
0810 18:01:42.376993 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.376997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.378199 (+ 1202us) server_negotiation.cc:407] Connection header received
0810 18:01:42.379468 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.379472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.379542 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.379646 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:42.381513 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.382395 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.383778 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.384015 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.384158 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.386983 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:42.386989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:42.386996 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:42.387023 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:42.388882 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.389518 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.389522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.389524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.389623 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.390022 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.390025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.390027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.390220 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:42.390326 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.390855 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.390977 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:01:42.391985 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.377515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:45275)
0810 18:01:42.378003 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.378087 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.379160 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.379998 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.380011 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.380512 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:42.381287 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.381314 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.382523 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.382526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.383654 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.383664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.384218 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.384223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.384398 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.385047 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:42.385063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:42.386785 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:42.388993 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.388998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.389009 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.389363 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.389739 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.389743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.389745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.389910 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.390327 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:42.390334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:42.390623 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.390798 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.391129 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":154,"threads_started":1}
I20260810 18:01:42.393168 4808 heartbeater.cc:347] Connected to a master server at 127.28.83.126:37985
I20260810 18:01:42.393332 4808 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:42.393848 4808 heartbeater.cc:511] Master 127.28.83.126:37985 requested a full tablet report, sending...
I20260810 18:01:42.394910 4320 ts_manager.cc:194] Registered new tserver with Master: c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485)
I20260810 18:01:42.395697 4320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:45275
I20260810 18:01:42.402637 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.395468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:43633 (local address 127.28.83.126:37985)
0810 18:01:42.395566 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.395569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.395750 (+ 181us) server_negotiation.cc:407] Connection header received
0810 18:01:42.395866 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.395869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.395914 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.395990 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:42.396482 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.397070 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.398148 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.398377 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.398488 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.399603 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:42.399607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:42.399609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:42.399633 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:42.400973 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.401427 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.401433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.401437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.401516 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.401837 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.401842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.401844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.402010 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:42.402126 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.402378 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.402484 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:01:42.402675 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.395400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:43633)
0810 18:01:42.395649 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.395663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.395760 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.396099 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.396101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.396270 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:42.396368 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.396373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.397187 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.397190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.398033 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.398042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.398489 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.398493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.398585 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.399136 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:42.399156 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:42.399468 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:42.401083 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.401106 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.401108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.401309 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.401613 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.401625 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.401627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.401744 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.402112 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:42.402115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:42.402213 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.402321 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.402466 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260810 18:01:42.404846 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:01:42 dist-test-slave-srgv krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:01:42 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:42 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:01:42.429273 4371 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:42.423026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.28.83.126:37985)
0810 18:01:42.423182 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.423186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.423199 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:01:42.423241 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.423243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.423283 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.423360 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:42.424062 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.424578 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.425742 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.425956 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.426041 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.429125 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59304: BlockingRecv error: recv got EOF from 127.0.0.1:59304 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 18:01:42 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:42.439096 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.430980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.28.83.126:37985)
0810 18:01:42.431139 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.431144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.431278 (+ 134us) server_negotiation.cc:407] Connection header received
0810 18:01:42.431426 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.431431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.431473 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.431554 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:42.432154 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.432654 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.433446 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.433661 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.433774 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.435997 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:42.436000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:42.436001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:42.436023 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:42.437252 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.437745 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.437748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.437750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.437840 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.438168 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.438173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.438174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.438358 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:42.438517 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.438806 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.438927 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:01:42.442305 4320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:42.445359 4320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:42.460433 4834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.454961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:34485 (local address 127.0.0.1:56496)
0810 18:01:42.455382 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.455397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.455490 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.456042 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.456044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.456065 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:42.456155 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.456160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.457353 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.457358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.458990 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.459001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.459803 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.459807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.459931 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.459975 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.460118 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.460249 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":136,"thread_start_us":95,"threads_started":1}
I20260810 18:01:42.460427 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.455192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.28.83.67:34485)
0810 18:01:42.455585 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.455598 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.455620 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:01:42.455693 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.455698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.455904 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.456072 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:42.456249 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.457225 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.459131 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.459672 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.459806 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.459984 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.460161 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.460258 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":100,"threads_started":1}
I20260810 18:01:42.460474 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.454264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:42001 (local address 127.0.0.1:57254)
0810 18:01:42.454760 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.454816 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.454950 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.455801 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.455808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.455836 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:42.456025 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.456032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.457708 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.457712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.458943 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.458956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.460037 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.460041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.460130 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.460157 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.460212 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.460294 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":51,"thread_start_us":70,"threads_started":1}
I20260810 18:01:42.460858 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.454628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:35143 (local address 127.0.0.1:54926)
0810 18:01:42.455754 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.455768 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.455856 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.456294 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.456297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.456315 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:42.456418 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.456424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.457864 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.457867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.459275 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.459287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.460489 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.460495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.460592 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.460612 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.460678 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.460742 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":340,"thread_start_us":110,"threads_started":1}
I20260810 18:01:42.461215 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.454441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.28.83.66:42001)
0810 18:01:42.455260 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.455274 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.455299 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:01:42.455376 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.455381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.455600 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.456180 (+ 580us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:42.456330 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.457594 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.459081 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.459893 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.460581 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.460753 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.460800 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.460888 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":125,"threads_started":1}
I20260810 18:01:42.461499 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.455008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.28.83.65:35143)
0810 18:01:42.455720 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.455726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.455859 (+ 133us) server_negotiation.cc:407] Connection header received
0810 18:01:42.455968 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.455975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.456144 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.456278 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:42.456550 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.457731 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.459416 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.460176 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.460288 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.460429 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.461183 (+ 754us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.461297 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":96,"threads_started":1}
I20260810 18:01:42.463970 4741 tablet_service.cc:1467] Processing CreateTablet for tablet c6ed8dbf2c7144c0816b739ca83b8774 (DEFAULT_TABLE table=test-table [id=f9369f40fc0b4bebbf63349cf91512bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:42.463967 4460 tablet_service.cc:1467] Processing CreateTablet for tablet c6ed8dbf2c7144c0816b739ca83b8774 (DEFAULT_TABLE table=test-table [id=f9369f40fc0b4bebbf63349cf91512bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:42.464927 4460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed8dbf2c7144c0816b739ca83b8774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:42.464928 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed8dbf2c7144c0816b739ca83b8774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:42.470949 4837 tablet_bootstrap.cc:492] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: Bootstrap starting.
I20260810 18:01:42.473423 4837 tablet_bootstrap.cc:654] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:42.473785 4838 tablet_bootstrap.cc:492] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: Bootstrap starting.
I20260810 18:01:42.473822 4600 tablet_service.cc:1467] Processing CreateTablet for tablet c6ed8dbf2c7144c0816b739ca83b8774 (DEFAULT_TABLE table=test-table [id=f9369f40fc0b4bebbf63349cf91512bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:42.474335 4837 log.cc:824] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:42.474905 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed8dbf2c7144c0816b739ca83b8774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:42.475953 4838 tablet_bootstrap.cc:654] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:42.476509 4837 tablet_bootstrap.cc:492] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: No bootstrap required, opened a new log
I20260810 18:01:42.476724 4837 ts_tablet_manager.cc:1397] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260810 18:01:42.476821 4838 log.cc:824] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:42.479220 4838 tablet_bootstrap.cc:492] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: No bootstrap required, opened a new log
I20260810 18:01:42.479413 4838 ts_tablet_manager.cc:1397] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:01:42.481585 4837 raft_consensus.cc:348] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } }
I20260810 18:01:42.481866 4837 raft_consensus.cc:374] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:42.481922 4837 raft_consensus.cc:729] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362c009765cf463fb2d60af0f9556f30, State: Initialized, Role: FOLLOWER
I20260810 18:01:42.482394 4837 consensus_queue.cc:260] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } }
I20260810 18:01:42.482865 4841 tablet_bootstrap.cc:492] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: Bootstrap starting.
I20260810 18:01:42.483397 4837 ts_tablet_manager.cc:1428] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260810 18:01:42.483453 4838 raft_consensus.cc:348] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } }
I20260810 18:01:42.483695 4838 raft_consensus.cc:374] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:42.483747 4838 raft_consensus.cc:729] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dd48aa8ca948d19df8cd94ca7cdad1, State: Initialized, Role: FOLLOWER
I20260810 18:01:42.483767 4526 heartbeater.cc:503] Master 127.28.83.126:37985 was elected leader, sending a full tablet report...
I20260810 18:01:42.484164 4838 consensus_queue.cc:260] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } }
I20260810 18:01:42.484856 4808 heartbeater.cc:503] Master 127.28.83.126:37985 was elected leader, sending a full tablet report...
I20260810 18:01:42.485157 4838 ts_tablet_manager.cc:1428] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:01:42.485201 4841 tablet_bootstrap.cc:654] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:42.486093 4841 log.cc:824] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:42.488181 4841 tablet_bootstrap.cc:492] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: No bootstrap required, opened a new log
I20260810 18:01:42.488391 4841 ts_tablet_manager.cc:1397] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:01:42.491514 4841 raft_consensus.cc:348] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } }
I20260810 18:01:42.491765 4841 raft_consensus.cc:374] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:42.491830 4841 raft_consensus.cc:729] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac3e7a2044f40d88038add091b46dcf, State: Initialized, Role: FOLLOWER
I20260810 18:01:42.492317 4841 consensus_queue.cc:260] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } }
I20260810 18:01:42.493093 4667 heartbeater.cc:503] Master 127.28.83.126:37985 was elected leader, sending a full tablet report...
I20260810 18:01:42.493391 4841 ts_tablet_manager.cc:1428] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:01:42.583666 4843 raft_consensus.cc:482] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:42.583917 4843 raft_consensus.cc:504] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } }
I20260810 18:01:42.585176 4843 leader_election.cc:288] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001), 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143)
I20260810 18:01:42.590775 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.585588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:37703 (local address 127.28.83.66:42001)
0810 18:01:42.585723 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.585727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.585743 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:01:42.585842 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.585845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.585895 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.585983 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:42.586344 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.587110 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.589345 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.590013 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.590146 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.590201 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.590562 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.590641 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:01:42.590775 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.585602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:50297 (local address 127.28.83.65:35143)
0810 18:01:42.585746 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.585749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.586327 (+ 578us) server_negotiation.cc:407] Connection header received
0810 18:01:42.586411 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.586414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.586465 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.586543 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:42.586803 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.587467 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.589345 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.590013 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.590146 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.590201 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.590562 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.590644 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:01:42.590811 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.585499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:42001 (local address 127.28.83.67:37703)
0810 18:01:42.585624 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.585641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.585741 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.586039 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.586043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.586071 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:42.586203 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.586210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.587253 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.587256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.589195 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.589208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.590165 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.590169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.590380 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.590426 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.590517 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.590631 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":119}
I20260810 18:01:42.590828 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.585500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:35143 (local address 127.28.83.67:50297)
0810 18:01:42.586207 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.586221 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.586321 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.586557 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.586559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.586579 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:42.586685 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.586689 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.587607 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.587611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.589194 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.589208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.590165 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.590169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.590377 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.590423 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.590517 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.590625 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":508,"thread_start_us":146,"threads_started":1}
I20260810 18:01:42.591558 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed8dbf2c7144c0816b739ca83b8774" candidate_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac3e7a2044f40d88038add091b46dcf" is_pre_election: true
I20260810 18:01:42.591566 4480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed8dbf2c7144c0816b739ca83b8774" candidate_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362c009765cf463fb2d60af0f9556f30" is_pre_election: true
I20260810 18:01:42.591930 4620 raft_consensus.cc:2454] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dd48aa8ca948d19df8cd94ca7cdad1 in term 0.
I20260810 18:01:42.591930 4480 raft_consensus.cc:2454] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dd48aa8ca948d19df8cd94ca7cdad1 in term 0.
I20260810 18:01:42.592571 4692 leader_election.cc:302] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 362c009765cf463fb2d60af0f9556f30, 7ac3e7a2044f40d88038add091b46dcf, c3dd48aa8ca948d19df8cd94ca7cdad1; no voters:
I20260810 18:01:42.592937 4843 raft_consensus.cc:2781] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:42.593024 4843 raft_consensus.cc:482] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:42.593055 4843 raft_consensus.cc:3037] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:42.594244 4843 raft_consensus.cc:504] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } }
I20260810 18:01:42.594671 4843 leader_election.cc:288] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [CANDIDATE]: Term 1 election: Requested vote from peers 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001), 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143)
I20260810 18:01:42.595113 4480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed8dbf2c7144c0816b739ca83b8774" candidate_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362c009765cf463fb2d60af0f9556f30"
I20260810 18:01:42.595232 4480 raft_consensus.cc:3037] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:42.596452 4480 raft_consensus.cc:2454] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dd48aa8ca948d19df8cd94ca7cdad1 in term 1.
I20260810 18:01:42.596766 4692 leader_election.cc:302] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362c009765cf463fb2d60af0f9556f30, c3dd48aa8ca948d19df8cd94ca7cdad1; no voters:
I20260810 18:01:42.596753 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed8dbf2c7144c0816b739ca83b8774" candidate_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac3e7a2044f40d88038add091b46dcf"
I20260810 18:01:42.596892 4620 raft_consensus.cc:3037] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:42.596957 4843 raft_consensus.cc:2781] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:42.597222 4843 raft_consensus.cc:686] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 LEADER]: Becoming Leader. State: Replica: c3dd48aa8ca948d19df8cd94ca7cdad1, State: Running, Role: LEADER
I20260810 18:01:42.597674 4843 consensus_queue.cc:237] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } }
I20260810 18:01:42.598546 4620 raft_consensus.cc:2454] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dd48aa8ca948d19df8cd94ca7cdad1 in term 1.
I20260810 18:01:42.601228 4321 catalog_manager.cc:5495] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67). New cstate: current_term: 1 leader_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } health_report { overall_health: UNKNOWN } } }
I20260810 18:01:42.612506 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.608943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.28.83.67:34485)
0810 18:01:42.609068 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.609072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.609087 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:01:42.609206 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.609209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.609264 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.609376 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:42.609660 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.610168 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.611053 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.611275 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.611386 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.611670 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:42.612091 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:42.612228 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.612306 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.612363 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 18:01:42.618405 4620 raft_consensus.cc:1261] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Refusing update from remote peer c3dd48aa8ca948d19df8cd94ca7cdad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:42.618405 4480 raft_consensus.cc:1261] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Refusing update from remote peer c3dd48aa8ca948d19df8cd94ca7cdad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:42.619169 4847 consensus_queue.cc:1035] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:42.619436 4843 consensus_queue.cc:1035] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:42.625523 4850 mvcc.cc:205] Tried to move back new op lower bound from 7317032561117204480 to 7317032561043288064. Current Snapshot: MvccSnapshot[applied={T|T < 7317032561117204480}]
I20260810 18:01:42.627226 4852 mvcc.cc:205] Tried to move back new op lower bound from 7317032561117204480 to 7317032561043288064. Current Snapshot: MvccSnapshot[applied={T|T < 7317032561117204480}]
I20260810 18:01:42.628424 4851 mvcc.cc:205] Tried to move back new op lower bound from 7317032561117204480 to 7317032561043288064. Current Snapshot: MvccSnapshot[applied={T|T < 7317032561117204480}]
I20260810 18:01:42.636809 4321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:42.637012 4321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:42.639679 4321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf: Sending DeleteTablet for 3 replicas of tablet c6ed8dbf2c7144c0816b739ca83b8774
I20260810 18:01:42.640635 4741 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ed8dbf2c7144c0816b739ca83b8774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:42 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:56496
I20260810 18:01:42.640944 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ed8dbf2c7144c0816b739ca83b8774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:42 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:57254
I20260810 18:01:42.641338 4860 tablet_replica.cc:330] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: stopping tablet replica
I20260810 18:01:42.641453 4321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:42.641923 4321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:42.641999 4460 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ed8dbf2c7144c0816b739ca83b8774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:42 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:54926
I20260810 18:01:42.642472 4860 raft_consensus.cc:2229] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:42.642607 4862 tablet_replica.cc:330] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: stopping tablet replica
I20260810 18:01:42.642936 4860 raft_consensus.cc:2258] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:42.642975 4862 raft_consensus.cc:2229] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:42.643405 4862 raft_consensus.cc:2258] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:42.643889 4861 tablet_replica.cc:330] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: stopping tablet replica
I20260810 18:01:42.644228 4861 raft_consensus.cc:2229] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:42.644541 4861 raft_consensus.cc:2258] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:42.645181 4862 ts_tablet_manager.cc:1905] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:42.646134 4861 ts_tablet_manager.cc:1905] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:42.648218 4862 ts_tablet_manager.cc:1918] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:42.648416 4862 log.cc:1196] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/c6ed8dbf2c7144c0816b739ca83b8774
I20260810 18:01:42.648825 4862 ts_tablet_manager.cc:1939] T c6ed8dbf2c7144c0816b739ca83b8774 P 362c009765cf463fb2d60af0f9556f30: Deleting consensus metadata
I20260810 18:01:42.648910 4861 ts_tablet_manager.cc:1918] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:42.649014 4861 log.cc:1196] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/c6ed8dbf2c7144c0816b739ca83b8774
I20260810 18:01:42.649017 4460 tablet_service.cc:1467] Processing CreateTablet for tablet cd193baa626047f180650a561b34d5a9 (DEFAULT_TABLE table=test-table [id=24bbdf8cd6ad4f4099e2a95b05c26a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:42.649331 4460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd193baa626047f180650a561b34d5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:42.649434 4861 ts_tablet_manager.cc:1939] T c6ed8dbf2c7144c0816b739ca83b8774 P 7ac3e7a2044f40d88038add091b46dcf: Deleting consensus metadata
I20260810 18:01:42.650801 4305 catalog_manager.cc:4841] TS 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143): tablet c6ed8dbf2c7144c0816b739ca83b8774 (table test-table [id=f9369f40fc0b4bebbf63349cf91512bb]) successfully deleted
I20260810 18:01:42.651633 4837 tablet_bootstrap.cc:492] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: Bootstrap starting.
I20260810 18:01:42.652025 4306 catalog_manager.cc:4841] TS 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001): tablet c6ed8dbf2c7144c0816b739ca83b8774 (table test-table [id=f9369f40fc0b4bebbf63349cf91512bb]) successfully deleted
I20260810 18:01:42.652374 4741 tablet_service.cc:1467] Processing CreateTablet for tablet cd193baa626047f180650a561b34d5a9 (DEFAULT_TABLE table=test-table [id=24bbdf8cd6ad4f4099e2a95b05c26a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:42.652580 4837 tablet_bootstrap.cc:654] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:42.652663 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd193baa626047f180650a561b34d5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:42.653296 4600 tablet_service.cc:1467] Processing CreateTablet for tablet cd193baa626047f180650a561b34d5a9 (DEFAULT_TABLE table=test-table [id=24bbdf8cd6ad4f4099e2a95b05c26a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:42.653582 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd193baa626047f180650a561b34d5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:42.654500 4860 ts_tablet_manager.cc:1905] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:42.655448 4838 tablet_bootstrap.cc:492] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: Bootstrap starting.
I20260810 18:01:42.655747 4841 tablet_bootstrap.cc:492] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: Bootstrap starting.
I20260810 18:01:42.656538 4837 tablet_bootstrap.cc:492] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: No bootstrap required, opened a new log
I20260810 18:01:42.656639 4837 ts_tablet_manager.cc:1397] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260810 18:01:42.656731 4841 tablet_bootstrap.cc:654] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:42.657616 4837 raft_consensus.cc:348] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.657758 4837 raft_consensus.cc:374] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:42.657804 4837 raft_consensus.cc:729] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362c009765cf463fb2d60af0f9556f30, State: Initialized, Role: FOLLOWER
I20260810 18:01:42.657930 4837 consensus_queue.cc:260] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.658031 4841 tablet_bootstrap.cc:492] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: No bootstrap required, opened a new log
I20260810 18:01:42.658103 4841 ts_tablet_manager.cc:1397] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 18:01:42.658227 4837 ts_tablet_manager.cc:1428] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:42.658550 4841 raft_consensus.cc:348] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.658684 4841 raft_consensus.cc:374] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:42.658715 4841 raft_consensus.cc:729] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac3e7a2044f40d88038add091b46dcf, State: Initialized, Role: FOLLOWER
I20260810 18:01:42.658826 4841 consensus_queue.cc:260] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.659063 4841 ts_tablet_manager.cc:1428] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:01:42.659912 4838 tablet_bootstrap.cc:654] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: Neither blocks nor log segments found. Creating new log.
W20260810 18:01:42.660185 4668 tablet.cc:2369] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:42.661388 4838 tablet_bootstrap.cc:492] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: No bootstrap required, opened a new log
I20260810 18:01:42.661495 4838 ts_tablet_manager.cc:1397] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260810 18:01:42.661996 4838 raft_consensus.cc:348] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.662133 4838 raft_consensus.cc:374] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:42.662168 4838 raft_consensus.cc:729] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dd48aa8ca948d19df8cd94ca7cdad1, State: Initialized, Role: FOLLOWER
I20260810 18:01:42.662302 4838 consensus_queue.cc:260] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.662561 4838 ts_tablet_manager.cc:1428] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:01:42.665237 4860 ts_tablet_manager.cc:1918] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:42.665362 4860 log.cc:1196] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/c6ed8dbf2c7144c0816b739ca83b8774
I20260810 18:01:42.665714 4860 ts_tablet_manager.cc:1939] T c6ed8dbf2c7144c0816b739ca83b8774 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting consensus metadata
I20260810 18:01:42.666828 4306 catalog_manager.cc:4841] TS c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485): tablet c6ed8dbf2c7144c0816b739ca83b8774 (table test-table [id=f9369f40fc0b4bebbf63349cf91512bb]) successfully deleted
W20260810 18:01:42.685138 4527 tablet.cc:2369] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:42.710268 4842 raft_consensus.cc:482] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:42.710446 4842 raft_consensus.cc:504] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.711725 4842 leader_election.cc:288] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001), c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485)
I20260810 18:01:42.717042 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.712601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:37001 (local address 127.28.83.67:34485)
0810 18:01:42.712717 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.712723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.712882 (+ 159us) server_negotiation.cc:407] Connection header received
0810 18:01:42.713034 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.713037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.713087 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.713161 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:42.713488 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.714195 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.715693 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.716358 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.716493 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.716555 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.716820 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.716911 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:01:42.717059 4867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.712476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:34485 (local address 127.28.83.65:37001)
0810 18:01:42.712778 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.712789 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.712908 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.713203 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.713206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.713226 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:42.713347 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.713353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.714322 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.714325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.715566 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.715577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.716524 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.716528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.716650 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.716705 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.716784 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.716865 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":87,"thread_start_us":111,"threads_started":1}
I20260810 18:01:42.717092 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.712033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:42001 (local address 127.28.83.65:54627)
0810 18:01:42.712340 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.712358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.712478 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.713051 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.713054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.713067 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:42.713215 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.713222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.714263 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.714266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.715690 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.715701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.716625 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.716629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.716739 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.716780 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.716868 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.716930 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":77,"thread_start_us":109,"threads_started":1}
I20260810 18:01:42.717473 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.712122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:54627 (local address 127.28.83.66:42001)
0810 18:01:42.712278 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.712283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.712753 (+ 470us) server_negotiation.cc:407] Connection header received
0810 18:01:42.712815 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.712819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.712868 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.712953 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:42.713320 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.714135 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.715828 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.716402 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.716503 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.716555 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.717238 (+ 683us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.717345 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:01:42.717710 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd193baa626047f180650a561b34d5a9" candidate_uuid: "362c009765cf463fb2d60af0f9556f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" is_pre_election: true
I20260810 18:01:42.717916 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd193baa626047f180650a561b34d5a9" candidate_uuid: "362c009765cf463fb2d60af0f9556f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac3e7a2044f40d88038add091b46dcf" is_pre_election: true
I20260810 18:01:42.717978 4762 raft_consensus.cc:2454] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362c009765cf463fb2d60af0f9556f30 in term 0.
I20260810 18:01:42.718065 4620 raft_consensus.cc:2454] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362c009765cf463fb2d60af0f9556f30 in term 0.
I20260810 18:01:42.718402 4411 leader_election.cc:302] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362c009765cf463fb2d60af0f9556f30, c3dd48aa8ca948d19df8cd94ca7cdad1; no voters:
I20260810 18:01:42.718683 4842 raft_consensus.cc:2781] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:42.718750 4842 raft_consensus.cc:482] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:42.718788 4842 raft_consensus.cc:3037] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:42.720026 4842 raft_consensus.cc:504] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.720389 4842 leader_election.cc:288] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [CANDIDATE]: Term 1 election: Requested vote from peers 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001), c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485)
I20260810 18:01:42.720798 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd193baa626047f180650a561b34d5a9" candidate_uuid: "362c009765cf463fb2d60af0f9556f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1"
I20260810 18:01:42.720798 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd193baa626047f180650a561b34d5a9" candidate_uuid: "362c009765cf463fb2d60af0f9556f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac3e7a2044f40d88038add091b46dcf"
I20260810 18:01:42.720921 4762 raft_consensus.cc:3037] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:42.720921 4620 raft_consensus.cc:3037] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:42.722185 4620 raft_consensus.cc:2454] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362c009765cf463fb2d60af0f9556f30 in term 1.
I20260810 18:01:42.722184 4762 raft_consensus.cc:2454] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362c009765cf463fb2d60af0f9556f30 in term 1.
I20260810 18:01:42.722465 4411 leader_election.cc:302] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362c009765cf463fb2d60af0f9556f30, 7ac3e7a2044f40d88038add091b46dcf; no voters:
I20260810 18:01:42.722653 4842 raft_consensus.cc:2781] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:42.722890 4842 raft_consensus.cc:686] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 1 LEADER]: Becoming Leader. State: Replica: 362c009765cf463fb2d60af0f9556f30, State: Running, Role: LEADER
I20260810 18:01:42.723145 4842 consensus_queue.cc:237] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.725942 4321 catalog_manager.cc:5495] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 362c009765cf463fb2d60af0f9556f30 (127.28.83.65). New cstate: current_term: 1 leader_uuid: "362c009765cf463fb2d60af0f9556f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } health_report { overall_health: UNKNOWN } } }
Aug 10 18:01:42 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384901, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.126@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:42.772733 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.763507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.28.83.126:37985)
0810 18:01:42.763639 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:42.763643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:42.763911 (+ 268us) server_negotiation.cc:407] Connection header received
0810 18:01:42.764100 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:42.764102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:42.764147 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:42.764228 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:42.764903 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.765481 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.766268 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.766491 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.766599 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.769231 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:42.769234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:42.769236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:42.769258 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:42.770749 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.771348 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.771350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.771351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.771422 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:42.771742 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:42.771744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:42.771745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:42.771916 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:42.771987 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:42.772374 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 18:01:42.772568 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":50}
I20260810 18:01:42.772743 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:42.763490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.0.0.1:59320)
0810 18:01:42.763818 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:42.763830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:42.763974 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:42.764278 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:42.764281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:42.764688 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:42.764776 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.764783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.765594 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.765600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.766141 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:42.766148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:42.766606 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:42.766609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:42.766719 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:42.767361 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:42.767373 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:42.769065 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:42.770873 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.770884 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.770891 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.771249 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.771521 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:42.771525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:42.771527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:42.771636 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:42.772029 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:42.772036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:42.772189 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 18:01:42.772327 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 18:01:42.772502 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":19,"thread_start_us":139,"threads_started":1}
I20260810 18:01:42.777261 4321 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:59320:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:01:42.778200 4321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:42.783555 4460 tablet_service.cc:1467] Processing CreateTablet for tablet 700999df880d4514baf53f4911e29d83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9268bca4b1744ea9e430ac30395cba6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:01:42.783681 4600 tablet_service.cc:1467] Processing CreateTablet for tablet 700999df880d4514baf53f4911e29d83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9268bca4b1744ea9e430ac30395cba6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:01:42.784062 4460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700999df880d4514baf53f4911e29d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:42.784251 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700999df880d4514baf53f4911e29d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:42.786790 4837 tablet_bootstrap.cc:492] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: Bootstrap starting.
I20260810 18:01:42.787339 4841 tablet_bootstrap.cc:492] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: Bootstrap starting.
I20260810 18:01:42.787317 4741 tablet_service.cc:1467] Processing CreateTablet for tablet 700999df880d4514baf53f4911e29d83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9268bca4b1744ea9e430ac30395cba6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:01:42.787707 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700999df880d4514baf53f4911e29d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:42.787827 4837 tablet_bootstrap.cc:654] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:42.788282 4841 tablet_bootstrap.cc:654] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:42.789690 4841 tablet_bootstrap.cc:492] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: No bootstrap required, opened a new log
I20260810 18:01:42.789788 4841 ts_tablet_manager.cc:1397] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260810 18:01:42.790247 4841 raft_consensus.cc:348] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.790375 4841 raft_consensus.cc:374] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:42.790412 4841 raft_consensus.cc:729] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac3e7a2044f40d88038add091b46dcf, State: Initialized, Role: FOLLOWER
I20260810 18:01:42.790544 4841 consensus_queue.cc:260] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.790817 4838 tablet_bootstrap.cc:492] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: Bootstrap starting.
I20260810 18:01:42.790866 4841 ts_tablet_manager.cc:1428] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:42.790853 4845 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:42.790999 4845 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:42.791805 4838 tablet_bootstrap.cc:654] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:42.793222 4838 tablet_bootstrap.cc:492] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: No bootstrap required, opened a new log
I20260810 18:01:42.793332 4838 ts_tablet_manager.cc:1397] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 18:01:42.793885 4838 raft_consensus.cc:348] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.794009 4838 raft_consensus.cc:374] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:42.794056 4838 raft_consensus.cc:729] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dd48aa8ca948d19df8cd94ca7cdad1, State: Initialized, Role: FOLLOWER
I20260810 18:01:42.794188 4838 consensus_queue.cc:260] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.794492 4837 tablet_bootstrap.cc:492] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: No bootstrap required, opened a new log
I20260810 18:01:42.794464 4847 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:42.794581 4837 ts_tablet_manager.cc:1397] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260810 18:01:42.794592 4847 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:42.794888 4838 ts_tablet_manager.cc:1428] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:01:42.795115 4837 raft_consensus.cc:348] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.795222 4837 raft_consensus.cc:374] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:42.795243 4837 raft_consensus.cc:729] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362c009765cf463fb2d60af0f9556f30, State: Initialized, Role: FOLLOWER
I20260810 18:01:42.795332 4837 consensus_queue.cc:260] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:42.795601 4837 ts_tablet_manager.cc:1428] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:42.795554 4842 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:42.795679 4842 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 18:01:42.890841 4809 tablet.cc:2369] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:01:43.003211 4847 raft_consensus.cc:482] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:43.003363 4847 raft_consensus.cc:504] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.003978 4847 leader_election.cc:288] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143), 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001)
I20260810 18:01:43.004515 4480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700999df880d4514baf53f4911e29d83" candidate_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362c009765cf463fb2d60af0f9556f30" is_pre_election: true
I20260810 18:01:43.004518 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700999df880d4514baf53f4911e29d83" candidate_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac3e7a2044f40d88038add091b46dcf" is_pre_election: true
I20260810 18:01:43.004684 4480 raft_consensus.cc:2454] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dd48aa8ca948d19df8cd94ca7cdad1 in term 0.
I20260810 18:01:43.004701 4620 raft_consensus.cc:2454] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dd48aa8ca948d19df8cd94ca7cdad1 in term 0.
I20260810 18:01:43.005030 4692 leader_election.cc:302] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362c009765cf463fb2d60af0f9556f30, c3dd48aa8ca948d19df8cd94ca7cdad1; no voters:
I20260810 18:01:43.005267 4847 raft_consensus.cc:2781] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:43.005334 4847 raft_consensus.cc:482] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:43.005363 4847 raft_consensus.cc:3037] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:43.006412 4847 raft_consensus.cc:504] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.006747 4847 leader_election.cc:288] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [CANDIDATE]: Term 1 election: Requested vote from peers 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143), 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001)
I20260810 18:01:43.007069 4480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700999df880d4514baf53f4911e29d83" candidate_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362c009765cf463fb2d60af0f9556f30"
I20260810 18:01:43.007126 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700999df880d4514baf53f4911e29d83" candidate_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac3e7a2044f40d88038add091b46dcf"
I20260810 18:01:43.007200 4480 raft_consensus.cc:3037] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:43.007243 4620 raft_consensus.cc:3037] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:43.008111 4620 raft_consensus.cc:2454] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dd48aa8ca948d19df8cd94ca7cdad1 in term 1.
I20260810 18:01:43.008122 4480 raft_consensus.cc:2454] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dd48aa8ca948d19df8cd94ca7cdad1 in term 1.
I20260810 18:01:43.008404 4693 leader_election.cc:302] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac3e7a2044f40d88038add091b46dcf, c3dd48aa8ca948d19df8cd94ca7cdad1; no voters:
I20260810 18:01:43.008562 4847 raft_consensus.cc:2781] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:43.008654 4847 raft_consensus.cc:686] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 LEADER]: Becoming Leader. State: Replica: c3dd48aa8ca948d19df8cd94ca7cdad1, State: Running, Role: LEADER
I20260810 18:01:43.008790 4847 consensus_queue.cc:237] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.009503 4857 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: TxnStatusTablet state changed. Reason: New leader c3dd48aa8ca948d19df8cd94ca7cdad1. Latest consensus state: current_term: 1 leader_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:43.009619 4857 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:01:43.009886 4880 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:01:43.010048 4880 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:01:43.010548 4319 catalog_manager.cc:5495] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67). New cstate: current_term: 1 leader_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } health_report { overall_health: HEALTHY } } }
I20260810 18:01:43.038991 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.035380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:34485 (local address 127.0.0.1:56508)
0810 18:01:43.035587 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:43.035601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:43.035692 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:43.035990 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:43.035992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:43.035998 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:43.036091 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.036096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.037068 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.037071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.037730 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.037740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.038321 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.038324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.038393 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.038441 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:43.038741 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:43.038752 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 18:01:43.038803 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 18:01:43.038875 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 18:01:43.039016 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.035504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.28.83.67:34485)
0810 18:01:43.035638 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.035643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.035660 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:01:43.035821 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.035824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.035878 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.035962 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:43.036193 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.036963 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.037851 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.038218 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.038322 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.038574 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:43.038660 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:43.038730 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:43.038830 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 18:01:43.038882 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:01:43.041586 4620 raft_consensus.cc:1261] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Refusing update from remote peer c3dd48aa8ca948d19df8cd94ca7cdad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:43.041601 4480 raft_consensus.cc:1261] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Refusing update from remote peer c3dd48aa8ca948d19df8cd94ca7cdad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:43.042040 4857 consensus_queue.cc:1035] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:43.042384 4847 consensus_queue.cc:1035] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:43.043721 4845 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: TxnStatusTablet state changed. Reason: New leader c3dd48aa8ca948d19df8cd94ca7cdad1. Latest consensus state: current_term: 1 leader_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:43.043838 4845 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:43.043777 4842 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: TxnStatusTablet state changed. Reason: New leader c3dd48aa8ca948d19df8cd94ca7cdad1. Latest consensus state: current_term: 1 leader_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:43.043891 4842 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:43.045866 4847 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:43.045992 4847 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:01:43.047658 4857 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:43.047771 4857 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P c3dd48aa8ca948d19df8cd94ca7cdad1: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:01:43.048525 4845 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:43.048643 4845 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P 7ac3e7a2044f40d88038add091b46dcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:43.050587 4842 tablet_replica.cc:439] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } } }
I20260810 18:01:43.050707 4842 tablet_replica.cc:442] T 700999df880d4514baf53f4911e29d83 P 362c009765cf463fb2d60af0f9556f30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:01:43.055498 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.051326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.28.83.65:35143)
0810 18:01:43.051441 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.051445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.051675 (+ 230us) server_negotiation.cc:407] Connection header received
0810 18:01:43.051768 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.051771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.051837 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.051963 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:43.052265 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.053025 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.053926 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.054269 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.054367 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.054593 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:43.055169 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:43.055283 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:43.055318 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 18:01:43.055372 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:01:43.066284 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.062693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:37847 (local address 127.28.83.67:34485)
0810 18:01:43.062838 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.062845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.063069 (+ 224us) server_negotiation.cc:407] Connection header received
0810 18:01:43.063274 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.063282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.063331 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.063410 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:43.063678 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.064239 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.065136 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.065363 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.065535 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.065848 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:43.065918 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:43.065998 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:43.066089 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 18:01:43.066142 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:01:43.066313 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.062589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:34485 (local address 127.28.83.65:37847)
0810 18:01:43.062982 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:43.062994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:43.063167 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:43.063431 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:43.063434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:43.063441 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:43.063569 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.063574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.064353 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.064358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.065008 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.065017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.065541 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.065545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.065657 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.065700 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:43.066002 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:43.066011 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:01:43.066079 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:01:43.066157 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":123,"threads_started":1}
I20260810 18:01:43.073184 4620 raft_consensus.cc:1261] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Refusing update from remote peer 362c009765cf463fb2d60af0f9556f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:43.073498 4762 raft_consensus.cc:1261] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Refusing update from remote peer 362c009765cf463fb2d60af0f9556f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:43.074092 4868 consensus_queue.cc:1035] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:43.074301 4842 consensus_queue.cc:1035] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:43.098424 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.094279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:40531 (local address 127.28.83.65:35143)
0810 18:01:43.094408 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.094413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.094604 (+ 191us) server_negotiation.cc:407] Connection header received
0810 18:01:43.094772 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.094775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.094826 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.094904 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:43.095190 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.096019 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.097279 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.097572 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.097655 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.097957 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:43.098045 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:43.098142 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:43.098233 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 18:01:43.098286 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:01:43.098433 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.094141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:35143 (local address 127.28.83.67:40531)
0810 18:01:43.094529 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:43.094545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:43.094684 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:43.094936 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:43.094939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:43.094947 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:01:43.095074 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.095080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.096159 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.096162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.097144 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.097156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.097676 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.097680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.097771 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.097825 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:01:43.098147 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:01:43.098156 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:01:43.098223 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 18:01:43.098305 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260810 18:01:43.125313 4321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:43.125509 4321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:43.127805 4321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf: Sending DeleteTablet for 3 replicas of tablet cd193baa626047f180650a561b34d5a9
I20260810 18:01:43.128381 4460 tablet_service.cc:1514] Processing DeleteTablet for tablet cd193baa626047f180650a561b34d5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:43 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:54926
I20260810 18:01:43.128571 4862 tablet_replica.cc:330] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: stopping tablet replica
I20260810 18:01:43.128722 4862 raft_consensus.cc:2229] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:43.128736 4741 tablet_service.cc:1514] Processing DeleteTablet for tablet cd193baa626047f180650a561b34d5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:43 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:56496
I20260810 18:01:43.128901 4860 tablet_replica.cc:330] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: stopping tablet replica
I20260810 18:01:43.129006 4860 raft_consensus.cc:2229] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:43.129029 4862 raft_consensus.cc:2258] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:43.129148 4860 raft_consensus.cc:2258] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:43.129208 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet cd193baa626047f180650a561b34d5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:43 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:57254
I20260810 18:01:43.129423 4861 tablet_replica.cc:330] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: stopping tablet replica
I20260810 18:01:43.129561 4861 raft_consensus.cc:2229] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:43.129689 4861 raft_consensus.cc:2258] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:43.130115 4862 ts_tablet_manager.cc:1905] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:43.130183 4860 ts_tablet_manager.cc:1905] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:43.131309 4861 ts_tablet_manager.cc:1905] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:43.134358 4860 ts_tablet_manager.cc:1918] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:43.134500 4860 log.cc:1196] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/cd193baa626047f180650a561b34d5a9
I20260810 18:01:43.134553 4861 ts_tablet_manager.cc:1918] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:43.134677 4861 log.cc:1196] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/cd193baa626047f180650a561b34d5a9
I20260810 18:01:43.134981 4860 ts_tablet_manager.cc:1939] T cd193baa626047f180650a561b34d5a9 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting consensus metadata
I20260810 18:01:43.134991 4861 ts_tablet_manager.cc:1939] T cd193baa626047f180650a561b34d5a9 P 7ac3e7a2044f40d88038add091b46dcf: Deleting consensus metadata
I20260810 18:01:43.135720 4306 catalog_manager.cc:4841] TS c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485): tablet cd193baa626047f180650a561b34d5a9 (table test-table [id=24bbdf8cd6ad4f4099e2a95b05c26a6b]) successfully deleted
I20260810 18:01:43.135895 4306 catalog_manager.cc:4841] TS 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001): tablet cd193baa626047f180650a561b34d5a9 (table test-table [id=24bbdf8cd6ad4f4099e2a95b05c26a6b]) successfully deleted
I20260810 18:01:43.137820 4862 ts_tablet_manager.cc:1918] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:43.137923 4862 log.cc:1196] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/cd193baa626047f180650a561b34d5a9
I20260810 18:01:43.138249 4862 ts_tablet_manager.cc:1939] T cd193baa626047f180650a561b34d5a9 P 362c009765cf463fb2d60af0f9556f30: Deleting consensus metadata
I20260810 18:01:43.138969 4305 catalog_manager.cc:4841] TS 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143): tablet cd193baa626047f180650a561b34d5a9 (table test-table [id=24bbdf8cd6ad4f4099e2a95b05c26a6b]) successfully deleted
Aug 10 18:01:43 dist-test-slave-srgv krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:01:43 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:43 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:01:43.167294 4371 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:43.159045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.28.83.126:37985)
0810 18:01:43.159189 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.159194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.159455 (+ 261us) server_negotiation.cc:407] Connection header received
0810 18:01:43.159667 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.159670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.159742 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.159796 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:43.160818 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.161668 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.162891 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.163235 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.163306 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.167168 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59336: BlockingRecv error: recv got EOF from 127.0.0.1:59336 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 18:01:43 dist-test-slave-srgv krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM
I20260810 18:01:43.180289 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.169314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.28.83.126:37985)
0810 18:01:43.169418 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.169438 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.169918 (+ 480us) server_negotiation.cc:407] Connection header received
0810 18:01:43.170081 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.170084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.170132 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.170181 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:43.171007 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.171776 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.172911 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.173229 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.173296 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.176324 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:43.176327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:43.176330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:43.176358 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:43.178231 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:43.178881 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:43.178885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:43.178888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:43.178978 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:43.179275 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:43.179278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:43.179280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:43.179517 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:43.179639 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:43.179998 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 18:01:43.180120 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:01:43.184157 4321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:01:43.184674 4321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:43.190018 4741 tablet_service.cc:1467] Processing CreateTablet for tablet b779b5881a3d4653a631dc33dd6d1af5 (DEFAULT_TABLE table=test-table [id=380e4a2566b3444b9679d10e8c53ec2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:43.190331 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b779b5881a3d4653a631dc33dd6d1af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:43.192939 4838 tablet_bootstrap.cc:492] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: Bootstrap starting.
I20260810 18:01:43.192924 4460 tablet_service.cc:1467] Processing CreateTablet for tablet b779b5881a3d4653a631dc33dd6d1af5 (DEFAULT_TABLE table=test-table [id=380e4a2566b3444b9679d10e8c53ec2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:43.193223 4460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b779b5881a3d4653a631dc33dd6d1af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:43.193900 4838 tablet_bootstrap.cc:654] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:43.195092 4838 tablet_bootstrap.cc:492] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: No bootstrap required, opened a new log
I20260810 18:01:43.195166 4838 ts_tablet_manager.cc:1397] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:43.195520 4838 raft_consensus.cc:348] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.195640 4838 raft_consensus.cc:374] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:43.195652 4837 tablet_bootstrap.cc:492] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: Bootstrap starting.
I20260810 18:01:43.195669 4838 raft_consensus.cc:729] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dd48aa8ca948d19df8cd94ca7cdad1, State: Initialized, Role: FOLLOWER
I20260810 18:01:43.195788 4838 consensus_queue.cc:260] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.195921 4600 tablet_service.cc:1467] Processing CreateTablet for tablet b779b5881a3d4653a631dc33dd6d1af5 (DEFAULT_TABLE table=test-table [id=380e4a2566b3444b9679d10e8c53ec2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:43.196028 4838 ts_tablet_manager.cc:1428] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:43.196245 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b779b5881a3d4653a631dc33dd6d1af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:43.196764 4837 tablet_bootstrap.cc:654] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:43.198544 4837 tablet_bootstrap.cc:492] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: No bootstrap required, opened a new log
I20260810 18:01:43.198623 4837 ts_tablet_manager.cc:1397] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 18:01:43.198815 4841 tablet_bootstrap.cc:492] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: Bootstrap starting.
I20260810 18:01:43.199120 4837 raft_consensus.cc:348] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.199249 4837 raft_consensus.cc:374] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:43.199295 4837 raft_consensus.cc:729] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362c009765cf463fb2d60af0f9556f30, State: Initialized, Role: FOLLOWER
I20260810 18:01:43.199435 4837 consensus_queue.cc:260] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.199693 4837 ts_tablet_manager.cc:1428] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:43.199792 4841 tablet_bootstrap.cc:654] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:43.201572 4841 tablet_bootstrap.cc:492] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: No bootstrap required, opened a new log
I20260810 18:01:43.201673 4841 ts_tablet_manager.cc:1397] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:43.202136 4841 raft_consensus.cc:348] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.202277 4841 raft_consensus.cc:374] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:43.202318 4841 raft_consensus.cc:729] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac3e7a2044f40d88038add091b46dcf, State: Initialized, Role: FOLLOWER
I20260810 18:01:43.202443 4841 consensus_queue.cc:260] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.202732 4841 ts_tablet_manager.cc:1428] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:01:43.330206 4845 raft_consensus.cc:482] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:43.330355 4845 raft_consensus.cc:504] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.331596 4845 leader_election.cc:288] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143), c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485)
I20260810 18:01:43.335803 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.331961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:34305 (local address 127.28.83.67:34485)
0810 18:01:43.332096 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.332103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.332403 (+ 300us) server_negotiation.cc:407] Connection header received
0810 18:01:43.332595 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.332600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.332661 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.332747 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:43.333034 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.333630 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.334751 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.335161 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.335286 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.335356 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:43.335585 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 18:01:43.335661 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:01:43.335803 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.331959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:52021 (local address 127.28.83.65:35143)
0810 18:01:43.332201 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.332220 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.332443 (+ 223us) server_negotiation.cc:407] Connection header received
0810 18:01:43.332507 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.332510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.332570 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.332658 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:01:43.332926 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.333556 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.334749 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.335167 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.335302 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.335363 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:43.335581 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 18:01:43.335663 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260810 18:01:43.335858 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.331860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:35143 (local address 127.28.83.66:52021)
0810 18:01:43.332271 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:43.332295 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:43.332417 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:43.332667 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:43.332670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:43.332686 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:43.332807 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.332812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.333683 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.333686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.334622 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.334631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.335293 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.335296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.335446 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.335469 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:01:43.335561 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:01:43.335653 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":55,"thread_start_us":79,"threads_started":1}
I20260810 18:01:43.335870 4912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.331870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:34485 (local address 127.28.83.66:34305)
0810 18:01:43.332288 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:43.332300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:43.332439 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:43.332759 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:43.332762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:43.332778 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:01:43.332902 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.332908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.333748 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.333751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.334630 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:43.334639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.335284 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.335288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:43.335454 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.335472 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:01:43.335561 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 18:01:43.335648 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":48,"thread_start_us":97,"threads_started":1}
I20260810 18:01:43.336587 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b779b5881a3d4653a631dc33dd6d1af5" candidate_uuid: "7ac3e7a2044f40d88038add091b46dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" is_pre_election: true
I20260810 18:01:43.336756 4762 raft_consensus.cc:2454] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac3e7a2044f40d88038add091b46dcf in term 0.
I20260810 18:01:43.336807 4480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b779b5881a3d4653a631dc33dd6d1af5" candidate_uuid: "7ac3e7a2044f40d88038add091b46dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362c009765cf463fb2d60af0f9556f30" is_pre_election: true
I20260810 18:01:43.336963 4480 raft_consensus.cc:2454] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac3e7a2044f40d88038add091b46dcf in term 0.
I20260810 18:01:43.337175 4552 leader_election.cc:302] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac3e7a2044f40d88038add091b46dcf, c3dd48aa8ca948d19df8cd94ca7cdad1; no voters:
I20260810 18:01:43.337508 4845 raft_consensus.cc:2781] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:43.337579 4845 raft_consensus.cc:482] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:43.337618 4845 raft_consensus.cc:3037] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:43.338697 4845 raft_consensus.cc:504] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.339097 4845 leader_election.cc:288] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [CANDIDATE]: Term 1 election: Requested vote from peers 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143), c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485)
I20260810 18:01:43.339450 4480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b779b5881a3d4653a631dc33dd6d1af5" candidate_uuid: "7ac3e7a2044f40d88038add091b46dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362c009765cf463fb2d60af0f9556f30"
I20260810 18:01:43.339555 4480 raft_consensus.cc:3037] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:43.340612 4480 raft_consensus.cc:2454] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac3e7a2044f40d88038add091b46dcf in term 1.
I20260810 18:01:43.340932 4551 leader_election.cc:302] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362c009765cf463fb2d60af0f9556f30, 7ac3e7a2044f40d88038add091b46dcf; no voters:
I20260810 18:01:43.340895 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b779b5881a3d4653a631dc33dd6d1af5" candidate_uuid: "7ac3e7a2044f40d88038add091b46dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1"
I20260810 18:01:43.341074 4762 raft_consensus.cc:3037] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:43.341118 4845 raft_consensus.cc:2781] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:43.341320 4845 raft_consensus.cc:686] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 1 LEADER]: Becoming Leader. State: Replica: 7ac3e7a2044f40d88038add091b46dcf, State: Running, Role: LEADER
I20260810 18:01:43.341603 4845 consensus_queue.cc:237] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.342267 4762 raft_consensus.cc:2454] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac3e7a2044f40d88038add091b46dcf in term 1.
I20260810 18:01:43.343890 4321 catalog_manager.cc:5495] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66). New cstate: current_term: 1 leader_uuid: "7ac3e7a2044f40d88038add091b46dcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } health_report { overall_health: UNKNOWN } } }
I20260810 18:01:43.355147 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.351562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.28.83.66:42001)
0810 18:01:43.351838 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.351845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.351856 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:01:43.351924 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.351925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.351975 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.352076 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:43.352312 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.352844 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.353803 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.354043 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.354141 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.354369 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:43.354824 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:43.354911 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:43.354967 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 18:01:43.355013 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":123,"threads_started":1}
I20260810 18:01:43.357697 4762 raft_consensus.cc:1261] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Refusing update from remote peer 7ac3e7a2044f40d88038add091b46dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:43.357734 4480 raft_consensus.cc:1261] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Refusing update from remote peer 7ac3e7a2044f40d88038add091b46dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:43.358289 4913 consensus_queue.cc:1035] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:43.358459 4845 consensus_queue.cc:1035] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:43.369292 4321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:43.369485 4321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:43.372710 4321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf: Sending DeleteTablet for 3 replicas of tablet b779b5881a3d4653a631dc33dd6d1af5
I20260810 18:01:43.373351 4460 tablet_service.cc:1514] Processing DeleteTablet for tablet b779b5881a3d4653a631dc33dd6d1af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:43 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:54926
I20260810 18:01:43.373497 4741 tablet_service.cc:1514] Processing DeleteTablet for tablet b779b5881a3d4653a631dc33dd6d1af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:43 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:56496
I20260810 18:01:43.373597 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet b779b5881a3d4653a631dc33dd6d1af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:43 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:57254
I20260810 18:01:43.373708 4860 tablet_replica.cc:330] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: stopping tablet replica
I20260810 18:01:43.373802 4862 tablet_replica.cc:330] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: stopping tablet replica
I20260810 18:01:43.373835 4860 raft_consensus.cc:2229] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:43.373932 4862 raft_consensus.cc:2229] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:43.373973 4860 raft_consensus.cc:2258] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:43.374068 4862 raft_consensus.cc:2258] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:43.374243 4861 tablet_replica.cc:330] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: stopping tablet replica
I20260810 18:01:43.374361 4861 raft_consensus.cc:2229] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:43.374528 4321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:01:43.374686 4861 raft_consensus.cc:2258] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:43.374902 4860 ts_tablet_manager.cc:1905] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 18:01:43.374960 4321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:01:43.375910 4861 ts_tablet_manager.cc:1905] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:43.377190 4860 ts_tablet_manager.cc:1918] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:43.377274 4860 log.cc:1196] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/b779b5881a3d4653a631dc33dd6d1af5
I20260810 18:01:43.377606 4860 ts_tablet_manager.cc:1939] T b779b5881a3d4653a631dc33dd6d1af5 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting consensus metadata
I20260810 18:01:43.378283 4306 catalog_manager.cc:4841] TS c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485): tablet b779b5881a3d4653a631dc33dd6d1af5 (table test-table [id=380e4a2566b3444b9679d10e8c53ec2e]) successfully deleted
I20260810 18:01:43.378408 4861 ts_tablet_manager.cc:1918] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:43.378484 4861 log.cc:1196] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/b779b5881a3d4653a631dc33dd6d1af5
I20260810 18:01:43.378746 4861 ts_tablet_manager.cc:1939] T b779b5881a3d4653a631dc33dd6d1af5 P 7ac3e7a2044f40d88038add091b46dcf: Deleting consensus metadata
I20260810 18:01:43.379889 4306 catalog_manager.cc:4841] TS 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001): tablet b779b5881a3d4653a631dc33dd6d1af5 (table test-table [id=380e4a2566b3444b9679d10e8c53ec2e]) successfully deleted
I20260810 18:01:43.380539 4862 ts_tablet_manager.cc:1905] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:43.382993 4862 ts_tablet_manager.cc:1918] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:01:43.383078 4862 log.cc:1196] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/b779b5881a3d4653a631dc33dd6d1af5
I20260810 18:01:43.383363 4862 ts_tablet_manager.cc:1939] T b779b5881a3d4653a631dc33dd6d1af5 P 362c009765cf463fb2d60af0f9556f30: Deleting consensus metadata
I20260810 18:01:43.384014 4305 catalog_manager.cc:4841] TS 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143): tablet b779b5881a3d4653a631dc33dd6d1af5 (table test-table [id=380e4a2566b3444b9679d10e8c53ec2e]) successfully deleted
I20260810 18:01:43.385890 4460 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcd7f26277d4747adbe46aa79a845a6 (DEFAULT_TABLE table=test-table [id=562e7f2811014d5492073f4fa71432f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:43.386152 4460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcd7f26277d4747adbe46aa79a845a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:43.386176 4600 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcd7f26277d4747adbe46aa79a845a6 (DEFAULT_TABLE table=test-table [id=562e7f2811014d5492073f4fa71432f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:43.386176 4741 tablet_service.cc:1467] Processing CreateTablet for tablet 0dcd7f26277d4747adbe46aa79a845a6 (DEFAULT_TABLE table=test-table [id=562e7f2811014d5492073f4fa71432f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:01:43.386416 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcd7f26277d4747adbe46aa79a845a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:43.386416 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dcd7f26277d4747adbe46aa79a845a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:43.388336 4837 tablet_bootstrap.cc:492] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30: Bootstrap starting.
I20260810 18:01:43.388758 4841 tablet_bootstrap.cc:492] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: Bootstrap starting.
I20260810 18:01:43.388809 4838 tablet_bootstrap.cc:492] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: Bootstrap starting.
I20260810 18:01:43.389137 4837 tablet_bootstrap.cc:654] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:43.389637 4838 tablet_bootstrap.cc:654] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:43.389730 4841 tablet_bootstrap.cc:654] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:43.390463 4837 tablet_bootstrap.cc:492] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30: No bootstrap required, opened a new log
I20260810 18:01:43.390537 4837 ts_tablet_manager.cc:1397] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:43.390934 4838 tablet_bootstrap.cc:492] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: No bootstrap required, opened a new log
I20260810 18:01:43.390882 4837 raft_consensus.cc:348] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.391002 4837 raft_consensus.cc:374] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:43.391058 4837 raft_consensus.cc:729] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362c009765cf463fb2d60af0f9556f30, State: Initialized, Role: FOLLOWER
I20260810 18:01:43.391062 4838 ts_tablet_manager.cc:1397] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:01:43.391160 4837 consensus_queue.cc:260] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.391409 4837 ts_tablet_manager.cc:1428] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:01:43.391490 4838 raft_consensus.cc:348] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.391623 4838 raft_consensus.cc:374] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:43.391662 4838 raft_consensus.cc:729] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dd48aa8ca948d19df8cd94ca7cdad1, State: Initialized, Role: FOLLOWER
I20260810 18:01:43.391796 4838 consensus_queue.cc:260] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.392035 4838 ts_tablet_manager.cc:1428] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:01:43.392284 4841 tablet_bootstrap.cc:492] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: No bootstrap required, opened a new log
I20260810 18:01:43.392370 4841 ts_tablet_manager.cc:1397] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 18:01:43.392753 4841 raft_consensus.cc:348] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.392866 4841 raft_consensus.cc:374] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:43.392899 4841 raft_consensus.cc:729] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac3e7a2044f40d88038add091b46dcf, State: Initialized, Role: FOLLOWER
I20260810 18:01:43.393033 4841 consensus_queue.cc:260] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.393321 4841 ts_tablet_manager.cc:1428] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:01:43.559391 4842 raft_consensus.cc:482] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:01:43.559544 4842 raft_consensus.cc:504] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.560078 4842 leader_election.cc:288] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001), c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485)
I20260810 18:01:43.560631 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcd7f26277d4747adbe46aa79a845a6" candidate_uuid: "362c009765cf463fb2d60af0f9556f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac3e7a2044f40d88038add091b46dcf" is_pre_election: true
I20260810 18:01:43.560663 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcd7f26277d4747adbe46aa79a845a6" candidate_uuid: "362c009765cf463fb2d60af0f9556f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" is_pre_election: true
I20260810 18:01:43.560812 4620 raft_consensus.cc:2454] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362c009765cf463fb2d60af0f9556f30 in term 0.
I20260810 18:01:43.560842 4762 raft_consensus.cc:2454] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362c009765cf463fb2d60af0f9556f30 in term 0.
I20260810 18:01:43.561179 4411 leader_election.cc:302] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362c009765cf463fb2d60af0f9556f30, 7ac3e7a2044f40d88038add091b46dcf; no voters:
I20260810 18:01:43.561430 4842 raft_consensus.cc:2781] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:01:43.561498 4842 raft_consensus.cc:482] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:01:43.561548 4842 raft_consensus.cc:3037] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:43.562619 4842 raft_consensus.cc:504] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.563022 4842 leader_election.cc:288] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [CANDIDATE]: Term 1 election: Requested vote from peers 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001), c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485)
I20260810 18:01:43.563398 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcd7f26277d4747adbe46aa79a845a6" candidate_uuid: "362c009765cf463fb2d60af0f9556f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac3e7a2044f40d88038add091b46dcf"
I20260810 18:01:43.563421 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dcd7f26277d4747adbe46aa79a845a6" candidate_uuid: "362c009765cf463fb2d60af0f9556f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1"
I20260810 18:01:43.563547 4620 raft_consensus.cc:3037] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:43.563558 4762 raft_consensus.cc:3037] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:43.564623 4762 raft_consensus.cc:2454] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362c009765cf463fb2d60af0f9556f30 in term 1.
I20260810 18:01:43.564625 4620 raft_consensus.cc:2454] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362c009765cf463fb2d60af0f9556f30 in term 1.
I20260810 18:01:43.564931 4411 leader_election.cc:302] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362c009765cf463fb2d60af0f9556f30, 7ac3e7a2044f40d88038add091b46dcf; no voters:
I20260810 18:01:43.565135 4842 raft_consensus.cc:2781] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:43.565251 4842 raft_consensus.cc:686] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 1 LEADER]: Becoming Leader. State: Replica: 362c009765cf463fb2d60af0f9556f30, State: Running, Role: LEADER
I20260810 18:01:43.565445 4842 consensus_queue.cc:237] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } }
I20260810 18:01:43.567157 4321 catalog_manager.cc:5495] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 362c009765cf463fb2d60af0f9556f30 (127.28.83.65). New cstate: current_term: 1 leader_uuid: "362c009765cf463fb2d60af0f9556f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362c009765cf463fb2d60af0f9556f30" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 } health_report { overall_health: UNKNOWN } } }
I20260810 18:01:43.591580 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.588732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.28.83.65:35143)
0810 18:01:43.588862 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:43.588865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:43.588881 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:01:43.589008 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:43.589011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:43.589055 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:43.589122 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:01:43.589339 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.589923 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.590689 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:43.590898 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:43.590997 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:43.591225 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:01:43.591295 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:01:43.591349 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:43.591428 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 18:01:43.591472 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:01:43.598450 4762 raft_consensus.cc:1261] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Refusing update from remote peer 362c009765cf463fb2d60af0f9556f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:43.598515 4620 raft_consensus.cc:1261] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Refusing update from remote peer 362c009765cf463fb2d60af0f9556f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:01:43.598929 4842 consensus_queue.cc:1035] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac3e7a2044f40d88038add091b46dcf" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:43.599112 4890 consensus_queue.cc:1035] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3dd48aa8ca948d19df8cd94ca7cdad1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:01:43.633220 4321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:43.633394 4321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:01:43.635350 4321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acbf32bf1543412e9c085d41df9be5bf: Sending DeleteTablet for 3 replicas of tablet 0dcd7f26277d4747adbe46aa79a845a6
I20260810 18:01:43.635922 4460 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcd7f26277d4747adbe46aa79a845a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:43 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:54926
I20260810 18:01:43.635954 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcd7f26277d4747adbe46aa79a845a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:43 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:57254
I20260810 18:01:43.636116 4862 tablet_replica.cc:330] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30: stopping tablet replica
I20260810 18:01:43.636184 4861 tablet_replica.cc:330] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: stopping tablet replica
I20260810 18:01:43.636246 4862 raft_consensus.cc:2229] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:01:43.636317 4861 raft_consensus.cc:2229] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:43.636437 4861 raft_consensus.cc:2258] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:43.636446 4862 raft_consensus.cc:2258] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:43.636914 4741 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dcd7f26277d4747adbe46aa79a845a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:01:43 UTC) from {username='oryx', principal='oryx/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:56496
I20260810 18:01:43.637110 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 4394
I20260810 18:01:43.637298 4862 ts_tablet_manager.cc:1905] T 0dcd7f26277d4747adbe46aa79a845a6 P 362c009765cf463fb2d60af0f9556f30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:43.637467 4861 ts_tablet_manager.cc:1905] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:43.638976 4860 tablet_replica.cc:330] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: stopping tablet replica
I20260810 18:01:43.639146 4860 raft_consensus.cc:2229] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:01:43.639307 4860 raft_consensus.cc:2258] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:01:43.640857 4860 ts_tablet_manager.cc:1905] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:01:43.640889 4861 ts_tablet_manager.cc:1918] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:43.640969 4861 log.cc:1196] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/0dcd7f26277d4747adbe46aa79a845a6
I20260810 18:01:43.641258 4861 ts_tablet_manager.cc:1939] T 0dcd7f26277d4747adbe46aa79a845a6 P 7ac3e7a2044f40d88038add091b46dcf: Deleting consensus metadata
I20260810 18:01:43.642513 4306 catalog_manager.cc:4841] TS 7ac3e7a2044f40d88038add091b46dcf (127.28.83.66:42001): tablet 0dcd7f26277d4747adbe46aa79a845a6 (table test-table [id=562e7f2811014d5492073f4fa71432f7]) successfully deleted
I20260810 18:01:43.643561 4860 ts_tablet_manager.cc:1918] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:01:43.643638 4860 log.cc:1196] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/0dcd7f26277d4747adbe46aa79a845a6
I20260810 18:01:43.643893 4860 ts_tablet_manager.cc:1939] T 0dcd7f26277d4747adbe46aa79a845a6 P c3dd48aa8ca948d19df8cd94ca7cdad1: Deleting consensus metadata
I20260810 18:01:43.644656 4306 catalog_manager.cc:4841] TS c3dd48aa8ca948d19df8cd94ca7cdad1 (127.28.83.67:34485): tablet 0dcd7f26277d4747adbe46aa79a845a6 (table test-table [id=562e7f2811014d5492073f4fa71432f7]) successfully deleted
W20260810 18:01:43.648748 4305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:35143 (error 108)
I20260810 18:01:43.648867 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 4535
I20260810 18:01:43.651405 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:43.650688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:35143 (local address 127.0.0.1:54946)
0810 18:01:43.651221 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:43.651306 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:35143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":60,"threads_started":1}
W20260810 18:01:43.651537 4305 catalog_manager.cc:4584] TS 362c009765cf463fb2d60af0f9556f30 (127.28.83.65:35143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0dcd7f26277d4747adbe46aa79a845a6: Network error: Client connection negotiation failed: client connection to 127.28.83.65:35143: connect: Connection refused (error 111)
I20260810 18:01:43.657833 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 4676
I20260810 18:01:43.668301 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 4289
2026-08-10T18:01:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3409 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:43 dist-test-slave-srgv krb5kdc[4934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:43 dist-test-slave-srgv krb5kdc[4934](info): set up 2 sockets
Aug 10 18:01:43 dist-test-slave-srgv krb5kdc[4934](info): commencing operation
krb5kdc: starting...
W20260810 18:01:44.740891 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:44 dist-test-slave-srgv krb5kdc[4934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:44.869040 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36107
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:36107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:44.980508 4947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:44.980795 4947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:44.980836 4947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:44.984931 4947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:44.985000 4947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:44.985021 4947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:44.985037 4947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:44.985102 4947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:44.990538 4947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:36107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:44.991672 4947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:44.992719 4947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:01:45.001502 4947 server_base.cc:1034] running on GCE node
W20260810 18:01:45.001461 4955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:45.001451 4953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:45.001446 4952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:45.002306 4947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:45.003477 4947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:45.004642 4947 hybrid_clock.cc:648] HybridClock initialized: now 1786384905004624 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T18:01:45Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1346 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:01:45 dist-test-slave-srgv krb5kdc[4960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:01:45 dist-test-slave-srgv krb5kdc[4960](info): set up 2 sockets
Aug 10 18:01:45 dist-test-slave-srgv krb5kdc[4960](info): commencing operation
krb5kdc: starting...
W20260810 18:01:46.064998 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:01:46 dist-test-slave-srgv krb5kdc[4960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:01:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:01:46.192840 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43629
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:43629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:46.305840 4973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:46.306216 4973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:46.306303 4973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:46.310427 4973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:46.310523 4973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:46.310561 4973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:46.310598 4973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:46.310637 4973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:46.316071 4973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:43629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:46.317245 4973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:46.318367 4973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:46.326990 4978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:46.327010 4981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:46.326990 4979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:46.327090 4973 server_base.cc:1034] running on GCE node
I20260810 18:01:46.327970 4973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:46.329049 4973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:46.330207 4973 hybrid_clock.cc:648] HybridClock initialized: now 1786384906330190 us; error 32 us; skew 500 ppm
Aug 10 18:01:46 dist-test-slave-srgv krb5kdc[4960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:46.332993 4973 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:46.333933 4973 webserver.cc:460] Webserver started at http://127.28.83.126:34065/ using document root <none> and password file <none>
I20260810 18:01:46.334590 4973 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:46.334697 4973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:46.334941 4973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:46.337013 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "471e927f3c3c42fbb43cb02d6f354bc6"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "53b87be059e25b0a4b50ccca7308e125"
server_key_iv: "a55ec4116b44d9894e101e57515c53e4"
server_key_version: "encryptionkey@0"
I20260810 18:01:46.337687 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "471e927f3c3c42fbb43cb02d6f354bc6"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "53b87be059e25b0a4b50ccca7308e125"
server_key_iv: "a55ec4116b44d9894e101e57515c53e4"
server_key_version: "encryptionkey@0"
I20260810 18:01:46.341562 4973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:01:46.343935 4988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:46.345206 4973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:46.345355 4973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "471e927f3c3c42fbb43cb02d6f354bc6"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "53b87be059e25b0a4b50ccca7308e125"
server_key_iv: "a55ec4116b44d9894e101e57515c53e4"
server_key_version: "encryptionkey@0"
I20260810 18:01:46.345551 4973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:46.362596 4973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:46.365716 4973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:46.365926 4973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:46.373636 4973 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:43629
I20260810 18:01:46.373661 5041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:43629 every 8 connection(s)
I20260810 18:01:46.374632 4973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:46.377368 5042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:01:46.379972 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 4973
I20260810 18:01:46.380113 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:01:46.380415 29005 external_mini_cluster.cc:1428] Setting key 799251ca73c87120617ae6e05922cb0f
I20260810 18:01:46.384402 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: Bootstrap starting.
I20260810 18:01:46.387140 5042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: Neither blocks nor log segments found. Creating new log.
I20260810 18:01:46.388006 5042 log.cc:824] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:46.390252 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: No bootstrap required, opened a new log
Aug 10 18:01:46 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:46.393954 5042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } }
I20260810 18:01:46.394256 5042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:01:46.394330 5042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471e927f3c3c42fbb43cb02d6f354bc6, State: Initialized, Role: FOLLOWER
I20260810 18:01:46.394861 5042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } }
I20260810 18:01:46.394994 5042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:46.395092 5042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:46.395185 5042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:01:46.396199 5042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } }
I20260810 18:01:46.396544 5042 leader_election.cc:302] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 471e927f3c3c42fbb43cb02d6f354bc6; no voters:
I20260810 18:01:46.396867 5042 leader_election.cc:288] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:01:46.396987 5047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:01:46.397207 5047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 1 LEADER]: Becoming Leader. State: Replica: 471e927f3c3c42fbb43cb02d6f354bc6, State: Running, Role: LEADER
I20260810 18:01:46.397264 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:46.381918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.28.83.126:43629)
0810 18:01:46.382536 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:46.382562 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:46.382611 (+ 49us) server_negotiation.cc:407] Connection header received
0810 18:01:46.383562 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:46.383590 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:46.384055 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:46.384455 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:46.385509 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.386539 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.387729 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.388261 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.388382 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:46.391346 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:46.391351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:46.391365 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:46.391393 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:46.393753 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:46.394299 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:46.394308 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:46.394329 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:46.394488 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:46.394795 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:46.394798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:46.394799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:46.395362 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:46.395577 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:46.395905 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 18:01:46.396217 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":138,"threads_started":1}
I20260810 18:01:46.397624 5047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } }
I20260810 18:01:46.398233 5042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:46.400285 5049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471e927f3c3c42fbb43cb02d6f354bc6. Latest consensus state: current_term: 1 leader_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } } }
I20260810 18:01:46.400398 5049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:46.400756 5048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } } }
I20260810 18:01:46.400827 5063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:46.401038 5048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:46.403625 5063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:46.409204 5063 catalog_manager.cc:1347] Generated new cluster ID: 200e08c62ff3437293442aaa301855d4
I20260810 18:01:46.409281 5063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:46.419503 5063 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:01:46.420838 5063 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:46.428913 5063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: Generated new TSK 0
I20260810 18:01:46.429713 5063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:01:46.465361 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43629
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:46.580379 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:46.580652 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:46.580695 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:46.585253 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:46.585340 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:46.585495 5069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:46.591154 5069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:46.592286 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:46.593364 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:46.603245 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:46.603331 5069 server_base.cc:1034] running on GCE node
W20260810 18:01:46.603256 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:46.603252 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:46.603906 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:46.604591 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:46.605785 5069 hybrid_clock.cc:648] HybridClock initialized: now 1786384906605773 us; error 30 us; skew 500 ppm
Aug 10 18:01:46 dist-test-slave-srgv krb5kdc[4960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:46.608691 5069 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:01:46.609668 5069 webserver.cc:460] Webserver started at http://127.28.83.65:36835/ using document root <none> and password file <none>
I20260810 18:01:46.610246 5069 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:46.610342 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:46.610579 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:46.612644 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "0429356727254e4d8086b679e22f24fd"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "093efad998b2d72217c89105ecf3d249"
server_key_iv: "6e46a4d81dc2ba570da6581f0ed34f39"
server_key_version: "encryptionkey@0"
I20260810 18:01:46.613245 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "0429356727254e4d8086b679e22f24fd"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "093efad998b2d72217c89105ecf3d249"
server_key_iv: "6e46a4d81dc2ba570da6581f0ed34f39"
server_key_version: "encryptionkey@0"
I20260810 18:01:46.616930 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:46.619597 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:46.620880 5069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:01:46.621019 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "0429356727254e4d8086b679e22f24fd"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "093efad998b2d72217c89105ecf3d249"
server_key_iv: "6e46a4d81dc2ba570da6581f0ed34f39"
server_key_version: "encryptionkey@0"
I20260810 18:01:46.621141 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:46.643843 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:46.647069 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:46.647293 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:46.647907 5069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:46.648792 5069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:46.648877 5069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:46.648931 5069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:46.648945 5069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:46.658465 5069 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:41193
I20260810 18:01:46.658489 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:41193 every 8 connection(s)
I20260810 18:01:46.659416 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:46.662971 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 5069
I20260810 18:01:46.663094 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:01:46.663388 29005 external_mini_cluster.cc:1428] Setting key 2314d0f3b298fd083de2bb2fc6d9f863
Aug 10 18:01:46 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:01:46.674201 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:46.661324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:58239 (local address 127.28.83.126:43629)
0810 18:01:46.661511 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:46.661516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:46.662441 (+ 925us) server_negotiation.cc:407] Connection header received
0810 18:01:46.663930 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:46.663935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:46.664001 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:46.664091 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:46.665516 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.666049 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.667226 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.667476 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.667595 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:46.670223 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:46.670226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:46.670228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:46.670254 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:46.671779 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:46.672342 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:46.672345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:46.672346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:46.672424 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:46.672850 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:46.672853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:46.672855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:46.673048 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:46.673165 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:46.673888 (+ 723us) server_negotiation.cc:299] Negotiation successful
0810 18:01:46.674028 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:01:46.675120 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:46.661798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:58239)
0810 18:01:46.662267 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:46.662315 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:46.663704 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:46.664314 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:46.664322 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:46.664827 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:46.665325 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:46.665339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.666197 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.666201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:46.667093 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:46.667103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.667646 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.667650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:46.667820 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:46.668470 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:46.668486 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:46.670031 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:46.671910 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:46.671916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:46.671932 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:46.672220 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:46.672519 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:46.672523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:46.672525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:46.672740 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:46.673166 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:46.673174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:46.673527 (+ 353us) client_negotiation.cc:769] Sending connection context
0810 18:01:46.673826 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 18:01:46.674189 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:01:46.676661 5199 heartbeater.cc:347] Connected to a master server at 127.28.83.126:43629
I20260810 18:01:46.676962 5199 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:46.677647 5199 heartbeater.cc:511] Master 127.28.83.126:43629 requested a full tablet report, sending...
I20260810 18:01:46.679467 5000 ts_manager.cc:194] Registered new tserver with Master: 0429356727254e4d8086b679e22f24fd (127.28.83.65:41193)
I20260810 18:01:46.681202 5000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:58239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:01:46.692984 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43629
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:46.808104 5204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:46.808384 5204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:46.808429 5204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:46.812556 5204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:46.812620 5204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:46.812678 5204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:46.817993 5204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:46.819022 5204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:46.820117 5204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:46.829746 5209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:46.829769 5210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:46.829952 5204 server_base.cc:1034] running on GCE node
W20260810 18:01:46.829759 5212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:46.830471 5204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:46.831104 5204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:46.832279 5204 hybrid_clock.cc:648] HybridClock initialized: now 1786384906832265 us; error 24 us; skew 500 ppm
Aug 10 18:01:46 dist-test-slave-srgv krb5kdc[4960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:46.835191 5204 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:01:46.835995 5204 webserver.cc:460] Webserver started at http://127.28.83.66:43963/ using document root <none> and password file <none>
I20260810 18:01:46.836512 5204 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:46.836575 5204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:46.836738 5204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:46.838788 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "0f1721780b3b4111a0f0e0ca64dcc7a6"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "05f73bd462c219e78627c69a295646d7"
server_key_iv: "e1623cb4eaf9e6485303e0190db28362"
server_key_version: "encryptionkey@0"
I20260810 18:01:46.839346 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "0f1721780b3b4111a0f0e0ca64dcc7a6"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "05f73bd462c219e78627c69a295646d7"
server_key_iv: "e1623cb4eaf9e6485303e0190db28362"
server_key_version: "encryptionkey@0"
I20260810 18:01:46.843264 5204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:01:46.845824 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:46.847121 5204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:46.847221 5204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "0f1721780b3b4111a0f0e0ca64dcc7a6"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "05f73bd462c219e78627c69a295646d7"
server_key_iv: "e1623cb4eaf9e6485303e0190db28362"
server_key_version: "encryptionkey@0"
I20260810 18:01:46.847312 5204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:46.858892 5204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:46.862340 5204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:46.862509 5204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:46.863051 5204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:46.863839 5204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:46.863888 5204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:46.863926 5204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:46.863948 5204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:46.873250 5204 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:36203
I20260810 18:01:46.873274 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:36203 every 8 connection(s)
I20260810 18:01:46.874260 5204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:46.880319 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 5204
I20260810 18:01:46.880458 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:01:46.880800 29005 external_mini_cluster.cc:1428] Setting key 2fdd11fe48e833cdac0decb0037c6cfd
Aug 10 18:01:46 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:46.889461 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:46.876090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:54583 (local address 127.28.83.126:43629)
0810 18:01:46.876246 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:46.876250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:46.877073 (+ 823us) server_negotiation.cc:407] Connection header received
0810 18:01:46.878469 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:46.878472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:46.878532 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:46.878646 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:46.879817 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.880352 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.881939 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.882221 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.882460 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:46.885169 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:46.885176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:46.885182 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:46.885216 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:46.887024 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:46.887584 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:46.887591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:46.887595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:46.887710 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:46.888064 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:46.888070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:46.888073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:46.888360 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:46.888489 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:46.889022 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 18:01:46.889183 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 18:01:46.890287 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:46.876450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:54583)
0810 18:01:46.876913 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:46.876956 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:46.878250 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:46.878815 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:46.878821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:46.879188 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:46.879659 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:46.879673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.880530 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.880534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:46.881813 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:46.881824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:46.882504 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:46.882507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:46.882673 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:46.883394 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:46.883411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:46.884989 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:46.887175 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:46.887180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:46.887195 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:46.887473 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:46.887818 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:46.887820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:46.887821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:46.887963 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:46.888477 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:46.888482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:46.888743 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 18:01:46.888907 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 18:01:46.889288 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260810 18:01:46.891597 5334 heartbeater.cc:347] Connected to a master server at 127.28.83.126:43629
I20260810 18:01:46.891893 5334 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
I20260810 18:01:46.892541 5334 heartbeater.cc:511] Master 127.28.83.126:43629 requested a full tablet report, sending...
I20260810 18:01:46.893707 5000 ts_manager.cc:194] Registered new tserver with Master: 0f1721780b3b4111a0f0e0ca64dcc7a6 (127.28.83.66:36203)
I20260810 18:01:46.894320 5000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:54583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:01:46.912070 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43629
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:47.025508 5339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:47.025894 5339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:47.025965 5339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:47.030267 5339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:47.030346 5339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:47.030441 5339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:47.035997 5339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:47.037177 5339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:47.038394 5339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:47.048302 5347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:47.048281 5344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:47.048421 5339 server_base.cc:1034] running on GCE node
W20260810 18:01:47.048281 5345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:47.049021 5339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:47.049837 5339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:47.051028 5339 hybrid_clock.cc:648] HybridClock initialized: now 1786384907050980 us; error 60 us; skew 500 ppm
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384907, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:47.054004 5339 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:01:47.054899 5339 webserver.cc:460] Webserver started at http://127.28.83.67:38989/ using document root <none> and password file <none>
I20260810 18:01:47.055503 5339 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:47.055601 5339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:47.055843 5339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:01:47.058003 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "b6a01d0babff4b82b5fad76f93e54da5"
format_stamp: "Formatted at 2026-08-10 18:01:47 on dist-test-slave-srgv"
server_key: "609b436f40f869279f683d707889e8de"
server_key_iv: "69c3dda15ef0732c2016475ee25533f6"
server_key_version: "encryptionkey@0"
I20260810 18:01:47.058629 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "b6a01d0babff4b82b5fad76f93e54da5"
format_stamp: "Formatted at 2026-08-10 18:01:47 on dist-test-slave-srgv"
server_key: "609b436f40f869279f683d707889e8de"
server_key_iv: "69c3dda15ef0732c2016475ee25533f6"
server_key_version: "encryptionkey@0"
I20260810 18:01:47.062526 5339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:01:47.065061 5354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.066399 5339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:47.066545 5339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "b6a01d0babff4b82b5fad76f93e54da5"
format_stamp: "Formatted at 2026-08-10 18:01:47 on dist-test-slave-srgv"
server_key: "609b436f40f869279f683d707889e8de"
server_key_iv: "69c3dda15ef0732c2016475ee25533f6"
server_key_version: "encryptionkey@0"
I20260810 18:01:47.066668 5339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:47.083609 5339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:47.086858 5339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:47.087085 5339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:47.087719 5339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:47.088584 5339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:47.088665 5339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.088738 5339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:47.088773 5339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.098441 5339 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:35515
I20260810 18:01:47.098475 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:35515 every 8 connection(s)
I20260810 18:01:47.099493 5339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384907, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:01:47.109463 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 5339
I20260810 18:01:47.109613 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:01:47.109915 29005 external_mini_cluster.cc:1428] Setting key 4ab169456ad2430db542175a52a3c2f4
I20260810 18:01:47.113890 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:47.101462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:35593 (local address 127.28.83.126:43629)
0810 18:01:47.101629 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:47.101634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:47.102553 (+ 919us) server_negotiation.cc:407] Connection header received
0810 18:01:47.103855 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:47.103858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:47.103907 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:47.103986 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:47.105181 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.105725 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.106833 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.107047 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.107131 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.109976 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:47.109980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:47.109982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:47.110032 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:47.111727 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:47.112301 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:47.112304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:47.112306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:47.112391 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:47.112750 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:47.112753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:47.112755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:47.112944 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:47.113053 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:47.113605 (+ 552us) server_negotiation.cc:299] Negotiation successful
0810 18:01:47.113728 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:01:47.114737 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:47.101912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:35593)
0810 18:01:47.102391 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:47.102435 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:47.103660 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:47.104207 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:47.104214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:47.104575 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:47.105047 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.105060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.105835 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.105838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.106718 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.106727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.107198 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.107201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.107339 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.108008 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:47.108024 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:47.109794 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:01:47.111845 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:47.111851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:47.111866 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:47.112205 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:47.112494 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:01:47.112497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:01:47.112499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:01:47.112652 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:01:47.113066 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:01:47.113075 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:01:47.113325 (+ 250us) client_negotiation.cc:769] Sending connection context
0810 18:01:47.113564 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 18:01:47.113913 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260810 18:01:47.115833 5469 heartbeater.cc:347] Connected to a master server at 127.28.83.126:43629
I20260810 18:01:47.116084 5469 heartbeater.cc:464] Registering TS with master...
I20260810 18:01:47.116577 5469 heartbeater.cc:511] Master 127.28.83.126:43629 requested a full tablet report, sending...
I20260810 18:01:47.117610 5000 ts_manager.cc:194] Registered new tserver with Master: b6a01d0babff4b82b5fad76f93e54da5 (127.28.83.67:35515)
I20260810 18:01:47.118182 5000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:35593
I20260810 18:01:47.124032 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab.
Entry for principal oryx/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab.
WARNING: no policy specified for oryx/127.28.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab.
Entry for principal oryx/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab.
WARNING: no policy specified for oryx/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab.
Entry for principal oryx/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab.
I20260810 18:01:47.200606 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 5069
I20260810 18:01:47.207334 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 5204
I20260810 18:01:47.214171 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 5339
I20260810 18:01:47.221196 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 4973
I20260810 18:01:47.228157 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43629
--webserver_interface=127.28.83.126
--webserver_port=34065
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:43629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:01:47.341766 5478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:47.342082 5478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:47.342123 5478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:47.346172 5478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:01:47.346233 5478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:47.346253 5478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:47.346271 5478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:01:47.346287 5478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:01:47.351642 5478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:43629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=34065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:47.352682 5478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:47.353747 5478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:47.362248 5483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:47.362433 5478 server_base.cc:1034] running on GCE node
W20260810 18:01:47.362273 5484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:47.362272 5486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:47.363216 5478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:47.364351 5478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:47.365559 5478 hybrid_clock.cc:648] HybridClock initialized: now 1786384907365547 us; error 38 us; skew 500 ppm
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384907, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:47.368350 5478 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:01:47.369231 5478 webserver.cc:460] Webserver started at http://127.28.83.126:34065/ using document root <none> and password file <none>
I20260810 18:01:47.369865 5478 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:47.369987 5478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:47.374035 5478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:01:47.376305 5494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.377616 5478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:47.377761 5478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "471e927f3c3c42fbb43cb02d6f354bc6"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "53b87be059e25b0a4b50ccca7308e125"
server_key_iv: "a55ec4116b44d9894e101e57515c53e4"
server_key_version: "encryptionkey@0"
I20260810 18:01:47.378357 5478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:47.399001 5478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:47.402586 5478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:47.402822 5478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:47.410641 5478 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:43629
I20260810 18:01:47.410724 5547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:43629 every 8 connection(s)
I20260810 18:01:47.411861 5478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:01:47.415464 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 5478
I20260810 18:01:47.415930 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:41193
--local_ip_for_outbound_sockets=127.28.83.65
--tserver_master_addrs=127.28.83.126:43629
--webserver_port=36835
--webserver_interface=127.28.83.65
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.83.65
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:01:47.416510 5548 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:01:47.418762 5548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: Bootstrap starting.
I20260810 18:01:47.443313 5548 log.cc:824] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:01:47.451799 5548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:01:47.452273 5548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: Bootstrap complete.
I20260810 18:01:47.458068 5548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } }
I20260810 18:01:47.458622 5548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471e927f3c3c42fbb43cb02d6f354bc6, State: Initialized, Role: FOLLOWER
I20260810 18:01:47.459185 5548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } }
I20260810 18:01:47.459332 5548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:01:47.459404 5548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:01:47.459501 5548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:01:47.462412 5548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } }
I20260810 18:01:47.462808 5548 leader_election.cc:302] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 471e927f3c3c42fbb43cb02d6f354bc6; no voters:
I20260810 18:01:47.463183 5548 leader_election.cc:288] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:01:47.463349 5552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:01:47.463600 5552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [term 2 LEADER]: Becoming Leader. State: Replica: 471e927f3c3c42fbb43cb02d6f354bc6, State: Running, Role: LEADER
I20260810 18:01:47.463922 5552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } }
I20260810 18:01:47.464542 5548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:01:47.465606 5553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } } }
I20260810 18:01:47.465735 5553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:47.466032 5554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471e927f3c3c42fbb43cb02d6f354bc6. Latest consensus state: current_term: 2 leader_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e927f3c3c42fbb43cb02d6f354bc6" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43629 } } }
I20260810 18:01:47.466125 5554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:01:47.467118 5561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:01:47.469853 5561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:01:47.470408 5561 catalog_manager.cc:1259] Loaded cluster ID: 200e08c62ff3437293442aaa301855d4
I20260810 18:01:47.470516 5561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:01:47.472251 5561 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:01:47.472832 5561 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 471e927f3c3c42fbb43cb02d6f354bc6: Loaded TSK: 0
I20260810 18:01:47.473896 5561 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:01:47.559746 5550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:47.560057 5550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:47.560130 5550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:47.564586 5550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:47.564694 5550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:47.564797 5550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:01:47.570839 5550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.65.keytab
--principal=oryx/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:41193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=36835
--tserver_master_addrs=127.28.83.126:43629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:47.571995 5550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:47.573086 5550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:47.583299 5576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:47.583441 5575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:47.583312 5578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:47.583650 5550 server_base.cc:1034] running on GCE node
I20260810 18:01:47.584311 5550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:47.585026 5550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:47.586223 5550 hybrid_clock.cc:648] HybridClock initialized: now 1786384907586178 us; error 68 us; skew 500 ppm
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384907, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:47.589442 5550 init.cc:377] Logged in from keytab as oryx/127.28.83.65@KRBTEST.COM (short username oryx)
I20260810 18:01:47.590566 5550 webserver.cc:460] Webserver started at http://127.28.83.65:36835/ using document root <none> and password file <none>
I20260810 18:01:47.591180 5550 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:47.591297 5550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:47.595692 5550 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:01:47.598017 5585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.599418 5550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:01:47.599527 5550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "0429356727254e4d8086b679e22f24fd"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "093efad998b2d72217c89105ecf3d249"
server_key_iv: "6e46a4d81dc2ba570da6581f0ed34f39"
server_key_version: "encryptionkey@0"
I20260810 18:01:47.600129 5550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:47.623458 5550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:47.627058 5550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:47.627316 5550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:47.628010 5550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:47.628938 5550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:47.628989 5550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.629060 5550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:47.629104 5550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.638990 5550 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:41193
I20260810 18:01:47.639072 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:41193 every 8 connection(s)
I20260810 18:01:47.640573 5550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:01:47.645136 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 5550
I20260810 18:01:47.645725 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:36203
--local_ip_for_outbound_sockets=127.28.83.66
--tserver_master_addrs=127.28.83.126:43629
--webserver_port=43963
--webserver_interface=127.28.83.66
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.83.66
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:47.654974 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:47.643116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:41169)
0810 18:01:47.643618 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:47.643678 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:47.645007 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:47.646479 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:47.646485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:47.646857 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:47.647337 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.647352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.648715 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.648718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.649704 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.649715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.650509 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.650513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.650716 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.651377 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:47.651395 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:47.653913 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":156,"threads_started":1}
W20260810 18:01:47.654989 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:47.642979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:41169 (local address 127.28.83.126:43629)
0810 18:01:47.643489 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:47.643507 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:47.643808 (+ 301us) server_negotiation.cc:407] Connection header received
0810 18:01:47.645538 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:47.645553 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:47.646003 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:47.646411 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:47.647541 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.648549 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.649837 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.650320 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.650512 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.654182 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:41169: BlockingRecv error: recv got EOF from 127.28.83.65:41169 (error 108)
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":148,"threads_started":1}
W20260810 18:01:47.655681 5700 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:43629 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.83.126:43629: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:47.661616 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:47.656191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:48371)
0810 18:01:47.656317 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:47.656333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:47.656440 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:47.656699 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:47.656702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:47.656871 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:47.656963 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.656968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.657786 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.657792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.658354 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.658361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.658840 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.658844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.658942 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.659495 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:47.659507 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:47.661467 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:01:47.661909 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:47.656304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:48371 (local address 127.28.83.126:43629)
0810 18:01:47.656434 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:47.656438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:47.656453 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:01:47.656535 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:47.656538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:47.656589 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:47.656683 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:47.657081 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.657676 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.658481 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.658715 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.658823 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.661633 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:48371: BlockingRecv error: recv got EOF from 127.28.83.65:48371 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:47.667793 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:47.662304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:39573)
0810 18:01:47.662426 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:47.662442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:47.662524 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:47.662794 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:47.662797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:47.662958 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:47.663031 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.663036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.663769 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.663774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.664317 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.664325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.664834 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.664838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.664918 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.665533 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:47.665552 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:47.667637 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 18:01:47.667902 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:47.662373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:39573 (local address 127.28.83.126:43629)
0810 18:01:47.662486 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:47.662491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:47.662504 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:01:47.662642 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:47.662645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:47.662692 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:47.662777 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:47.663155 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.663658 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.664477 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.664719 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.664832 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.667773 (+ 2941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:39573: BlockingRecv error: recv got EOF from 127.28.83.65:39573 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 18:01:47.668296 5700 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:01:47.763984 5704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:47.764344 5704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:47.764428 5704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:47.769014 5704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:47.769145 5704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:47.769246 5704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:01:47.774914 5704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.66.keytab
--principal=oryx/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:36203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=43963
--tserver_master_addrs=127.28.83.126:43629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:47.776094 5704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:47.777191 5704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:47.787230 5709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:47.787222 5710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:47.787222 5712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:47.787413 5704 server_base.cc:1034] running on GCE node
I20260810 18:01:47.788074 5704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:47.788803 5704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:47.790006 5704 hybrid_clock.cc:648] HybridClock initialized: now 1786384907789984 us; error 35 us; skew 500 ppm
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384907, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:47.792925 5704 init.cc:377] Logged in from keytab as oryx/127.28.83.66@KRBTEST.COM (short username oryx)
I20260810 18:01:47.793936 5704 webserver.cc:460] Webserver started at http://127.28.83.66:43963/ using document root <none> and password file <none>
I20260810 18:01:47.794551 5704 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:47.794674 5704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:47.798774 5704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:01:47.800963 5719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.802305 5704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:01:47.802457 5704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "0f1721780b3b4111a0f0e0ca64dcc7a6"
format_stamp: "Formatted at 2026-08-10 18:01:46 on dist-test-slave-srgv"
server_key: "05f73bd462c219e78627c69a295646d7"
server_key_iv: "e1623cb4eaf9e6485303e0190db28362"
server_key_version: "encryptionkey@0"
I20260810 18:01:47.803056 5704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:47.828378 5704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:47.831787 5704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:47.832026 5704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:47.832682 5704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:47.833626 5704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:47.833711 5704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.833810 5704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:47.833861 5704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:47.843197 5704 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:36203
I20260810 18:01:47.843227 5833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:36203 every 8 connection(s)
I20260810 18:01:47.844295 5704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:01:47.854084 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 5704
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:47.854614 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:35515
--local_ip_for_outbound_sockets=127.28.83.67
--tserver_master_addrs=127.28.83.126:43629
--webserver_port=38989
--webserver_interface=127.28.83.67
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.83.67
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:01:47.856207 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:47.846250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:42429 (local address 127.28.83.126:43629)
0810 18:01:47.846401 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:47.846404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:47.847326 (+ 922us) server_negotiation.cc:407] Connection header received
0810 18:01:47.848713 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:47.848716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:47.848781 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:47.848898 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:47.850216 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.850755 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.851912 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.852145 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.852261 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.856078 (+ 3817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:42429: BlockingRecv error: recv got EOF from 127.28.83.66:42429 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:01:47.856901 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:47.846643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:42429)
0810 18:01:47.847159 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:47.847206 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:47.848489 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:47.849088 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:47.849101 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:47.849563 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:47.850054 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.850068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.850901 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.850905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.851780 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.851790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.852310 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.852314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.852488 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.853133 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:47.853150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:47.855940 (+ 2790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
W20260810 18:01:47.857483 5834 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:43629 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.83.126:43629: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:47.864149 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:47.858018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:39019)
0810 18:01:47.858219 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:47.858236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:47.858322 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:47.858589 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:47.858595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:47.858798 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:47.858956 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.858968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.859699 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.859706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.860762 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.860777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.861260 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.861267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.861400 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.861946 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:47.861967 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:47.864025 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260810 18:01:47.864527 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:47.858134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:39019 (local address 127.28.83.126:43629)
0810 18:01:47.858255 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:47.858258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:47.858277 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:01:47.858418 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:47.858420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:47.858459 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:47.858532 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:47.859095 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.859585 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.860904 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.861134 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.861218 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.864232 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:39019: BlockingRecv error: recv got EOF from 127.28.83.66:39019 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:47 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:47.870404 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:47.864833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:53711)
0810 18:01:47.865036 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:47.865054 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:47.865158 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:47.865611 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:47.865614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:47.865825 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:47.865897 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.865902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.867003 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.867006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.867506 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:47.867513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.868073 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.868075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:47.868133 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.868631 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:47.868642 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:47.870310 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260810 18:01:47.870579 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:47.864947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:53711 (local address 127.28.83.126:43629)
0810 18:01:47.865088 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:47.865095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:47.865116 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:01:47.865253 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:47.865258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:47.865318 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:47.865433 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:47.866033 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.866893 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.867620 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:47.867964 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:47.868067 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:47.870454 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:53711: BlockingRecv error: recv got EOF from 127.28.83.66:53711 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 18:01:47.870805 5834 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:01:47.974153 5837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:01:47.974468 5837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:01:47.974534 5837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:01:47.978695 5837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:01:47.978775 5837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:01:47.978833 5837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:01:47.984416 5837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/krb5kdc/oryx_127.28.83.67.keytab
--principal=oryx/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:35515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=38989
--tserver_master_addrs=127.28.83.126:43629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:01:47.985546 5837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:01:47.986542 5837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:01:47.996572 5842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:47.996601 5843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:01:47.996572 5845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:01:47.996819 5837 server_base.cc:1034] running on GCE node
I20260810 18:01:47.997349 5837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:01:47.998060 5837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:01:47.999233 5837 hybrid_clock.cc:648] HybridClock initialized: now 1786384907999205 us; error 51 us; skew 500 ppm
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384908, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:01:48.002134 5837 init.cc:377] Logged in from keytab as oryx/127.28.83.67@KRBTEST.COM (short username oryx)
I20260810 18:01:48.003031 5837 webserver.cc:460] Webserver started at http://127.28.83.67:38989/ using document root <none> and password file <none>
I20260810 18:01:48.003613 5837 fs_manager.cc:359] Metadata directory not provided
I20260810 18:01:48.003738 5837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:01:48.008014 5837 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:01:48.010162 5852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:48.011647 5837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:01:48.011793 5837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "b6a01d0babff4b82b5fad76f93e54da5"
format_stamp: "Formatted at 2026-08-10 18:01:47 on dist-test-slave-srgv"
server_key: "609b436f40f869279f683d707889e8de"
server_key_iv: "69c3dda15ef0732c2016475ee25533f6"
server_key_version: "encryptionkey@0"
I20260810 18:01:48.012583 5837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:01:48.027115 5837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:01:48.030239 5837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:01:48.030447 5837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:01:48.031075 5837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:01:48.031950 5837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:01:48.031999 5837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:48.032040 5837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:01:48.032056 5837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:01:48.042016 5837 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:35515
I20260810 18:01:48.042045 5968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:35515 every 8 connection(s)
I20260810 18:01:48.043100 5837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:01:48.052706 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 5837
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:01:48.054935 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:48.045111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:49403 (local address 127.28.83.126:43629)
0810 18:01:48.045288 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:48.045291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:48.046143 (+ 852us) server_negotiation.cc:407] Connection header received
0810 18:01:48.047457 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:48.047460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:48.047518 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:48.047601 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:48.048867 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.049479 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.050656 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.050880 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.050971 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.054769 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:49403: BlockingRecv error: recv got EOF from 127.28.83.67:49403 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 18:01:48.055475 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:48.045462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:49403)
0810 18:01:48.045951 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:48.045998 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:48.047236 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:48.047790 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:48.047796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:48.048175 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:48.048659 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.048683 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.049610 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.049613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.050518 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.050527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.051029 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.051032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.051181 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.051791 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:48.051807 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:48.054592 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":123,"threads_started":1}
W20260810 18:01:48.056028 5969 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:43629 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.83.126:43629: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:48.061873 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:48.056534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:43947)
0810 18:01:48.056702 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:48.056715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:48.056786 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:48.057025 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:48.057029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:48.057212 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:48.057305 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.057310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.058077 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.058082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.058631 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.058639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.059108 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.059111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.059212 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.059801 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:48.059820 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:48.061716 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260810 18:01:48.061960 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:48.056631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:43947 (local address 127.28.83.126:43629)
0810 18:01:48.056741 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:48.056744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:48.056756 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:01:48.056865 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:48.056867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:48.056902 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:48.056975 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:48.057459 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.057961 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.058769 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.058990 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.059093 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.061834 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:43947: BlockingRecv error: recv got EOF from 127.28.83.67:43947 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:01:48.063301 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:48.053862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.28.83.126:43629)
0810 18:01:48.054145 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:48.054148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:48.054159 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:01:48.054191 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:48.054193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:48.054231 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:48.054333 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:48.055288 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.055919 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.057114 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.057325 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.057460 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.058982 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:01:48.058987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:01:48.059004 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:01:48.059031 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:01:48.060864 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:48.061330 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:48.061337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:48.061344 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:48.061552 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:01:48.061844 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:01:48.061848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:01:48.061850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:01:48.062323 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:01:48.062528 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 18:01:48.062876 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 18:01:48.063153 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:48.068255 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:48.062725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:52141)
0810 18:01:48.062903 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:48.062917 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:48.062994 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:48.063302 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:48.063305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:48.063467 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:48.063559 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.063566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.064262 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.064266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.065032 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.065043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.065659 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.065664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.065752 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.066261 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:48.066275 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:48.068158 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260810 18:01:48.068444 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:48.062804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:52141 (local address 127.28.83.126:43629)
0810 18:01:48.062934 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:48.062937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:48.062953 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:01:48.063143 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:48.063146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:48.063194 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:48.063287 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:48.063676 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.064153 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.065179 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.065380 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.065542 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.068305 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:52141: BlockingRecv error: recv got EOF from 127.28.83.67:52141 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 18:01:48.068735 5969 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:48.674978 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:48.669068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:53415)
0810 18:01:48.669456 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:48.669473 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:48.669594 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:48.669910 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:48.669913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:48.670191 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:48.670319 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.670325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.671194 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.671197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.671803 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.671812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.672228 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.672231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.672307 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.672836 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:48.672848 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:48.674844 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":94,"threads_started":1}
W20260810 18:01:48.675232 5975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:48.669218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:53415 (local address 127.28.83.126:43629)
0810 18:01:48.669555 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:48.669559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:48.669578 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:01:48.669700 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:48.669703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:48.669763 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:48.669876 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:48.670408 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.671076 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.671910 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.672121 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.672193 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.675061 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:53415: BlockingRecv error: recv got EOF from 127.28.83.65:53415 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:48 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:48.877260 5976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:48.871457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:33035)
0810 18:01:48.871753 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:48.871767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:48.871907 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:48.872232 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:48.872235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:48.872458 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:48.872552 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.872557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.873336 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.873341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.873956 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:48.873965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.874457 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.874461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:48.874560 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.875118 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:48.875133 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:48.877132 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":90,"threads_started":1}
W20260810 18:01:48.877497 5975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:48.871520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:33035 (local address 127.28.83.126:43629)
0810 18:01:48.871694 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:48.871699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:48.871852 (+ 153us) server_negotiation.cc:407] Connection header received
0810 18:01:48.872014 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:48.872018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:48.872074 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:48.872206 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:48.872683 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.873214 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.874089 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:48.874326 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:48.874452 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:48.877325 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:33035: BlockingRecv error: recv got EOF from 127.28.83.66:33035 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 18:01:49 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:49 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:49.075531 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:49.069398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:52423)
0810 18:01:49.069727 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:49.069765 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:49.069894 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:49.070243 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:49.070246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:49.070457 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:49.070553 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:49.070559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.071379 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.071384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:49.071977 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:49.071985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.072507 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.072511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:49.072600 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:49.073283 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:49.073295 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:49.075343 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":89,"threads_started":1}
W20260810 18:01:49.075615 5975 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:49.069506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:52423 (local address 127.28.83.126:43629)
0810 18:01:49.069688 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:49.069692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:49.069854 (+ 162us) server_negotiation.cc:407] Connection header received
0810 18:01:49.069995 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:49.070001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:49.070084 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:49.070189 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:49.070692 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.071256 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.072108 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.072378 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.072485 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:49.075492 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:52423: BlockingRecv error: recv got EOF from 127.28.83.67:52423 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 18:01:49 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:49 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:49.681181 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:49.675850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:56949)
0810 18:01:49.676163 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:49.676177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:49.676274 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:49.676508 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:49.676510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:49.676697 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:49.676796 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:49.676801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.677537 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.677540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:49.678100 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:49.678107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.678525 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.678528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:49.678589 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:49.679114 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:49.679127 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:49.681061 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
W20260810 18:01:49.681485 5979 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:49.675978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:56949 (local address 127.28.83.126:43629)
0810 18:01:49.676272 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:49.676275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:49.676289 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:01:49.676343 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:49.676345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:49.676387 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:49.676463 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:49.676904 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.677448 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.678212 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.678428 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.678497 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:49.681199 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:56949: BlockingRecv error: recv got EOF from 127.28.83.65:56949 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":124,"threads_started":1}
Aug 10 18:01:49 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:49 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:49.884136 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:49.878113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:46423)
0810 18:01:49.878438 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:49.878453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:49.878594 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:49.878892 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:49.878896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:49.879111 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:49.879205 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:49.879222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.880002 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.880008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:49.880568 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:49.880576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.881073 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.881076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:49.881160 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:49.881816 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:49.881828 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:49.883854 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
W20260810 18:01:49.884224 5979 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:49.878190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:46423 (local address 127.28.83.126:43629)
0810 18:01:49.878378 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:49.878383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:49.878534 (+ 151us) server_negotiation.cc:407] Connection header received
0810 18:01:49.878701 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:49.878704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:49.878771 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:49.878863 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:49.879344 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.879882 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.880698 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:49.880945 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:49.881057 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:49.884098 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:46423: BlockingRecv error: recv got EOF from 127.28.83.66:46423 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 18:01:50 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:50 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:50.087399 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:50.076492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:45069)
0810 18:01:50.076797 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:50.076811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:50.076904 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:50.077200 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:50.077202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:50.077388 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:50.077567 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:50.077574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.078319 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.078322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:50.078876 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:50.078885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.079331 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.079333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:50.079395 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:50.085109 (+ 5714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:50.085122 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:50.087185 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":121,"threads_started":1}
W20260810 18:01:50.087441 5979 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:50.076682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:45069 (local address 127.28.83.126:43629)
0810 18:01:50.076833 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:50.076837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:50.076854 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:01:50.077003 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:50.077005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:50.077081 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:50.077159 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:50.077680 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.078221 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.078982 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.079223 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.079303 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:50.087334 (+ 8031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:45069: BlockingRecv error: recv got EOF from 127.28.83.67:45069 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 18:01:50 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:50 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:50.688023 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:50.682038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:45123)
0810 18:01:50.682326 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:50.682343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:50.682462 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:50.682773 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:50.682775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:50.682970 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:50.683075 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:50.683088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.683919 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.683922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:50.684555 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:50.684564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.685069 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.685071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:50.685155 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:50.685811 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:50.685824 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:50.687888 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
W20260810 18:01:50.688197 5983 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:50.682115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:45123 (local address 127.28.83.126:43629)
0810 18:01:50.682410 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:50.682414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:50.682430 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:01:50.682585 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:50.682587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:50.682635 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:50.682724 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:50.683190 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.683809 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.684722 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.684951 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.685041 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:50.688053 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:45123: BlockingRecv error: recv got EOF from 127.28.83.65:45123 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":156,"threads_started":1}
Aug 10 18:01:50 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:50 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:50.890975 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:50.885119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:40587)
0810 18:01:50.885483 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:50.885498 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:50.885632 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:50.885927 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:50.885931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:50.886163 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:50.886258 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:50.886264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.887070 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.887076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:50.887664 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:50.887672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.888165 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.888169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:50.888257 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:50.888775 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:50.888787 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:50.890792 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
W20260810 18:01:50.891052 5983 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:50.885230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:40587 (local address 127.28.83.126:43629)
0810 18:01:50.885428 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:50.885434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:50.885577 (+ 143us) server_negotiation.cc:407] Connection header received
0810 18:01:50.885747 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:50.885750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:50.885806 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:50.885900 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:50.886390 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.886949 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.887797 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:50.888037 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:50.888157 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:50.890937 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:40587: BlockingRecv error: recv got EOF from 127.28.83.66:40587 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 18:01:51 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:51 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:51.094254 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:51.088428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:55493)
0810 18:01:51.088797 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:51.088811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:51.088912 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:51.089186 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:51.089188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:51.089382 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:51.089553 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:51.089559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.090302 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.090305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:51.090858 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:51.090866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.091298 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.091301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:51.091382 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:51.091910 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:51.091921 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:51.094038 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
W20260810 18:01:51.094292 5983 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:51.088565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:55493 (local address 127.28.83.126:43629)
0810 18:01:51.088710 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:51.088714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:51.088883 (+ 169us) server_negotiation.cc:407] Connection header received
0810 18:01:51.088996 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:51.088998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:51.089045 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:51.089148 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:51.089665 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.090205 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.090963 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.091198 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.091275 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:51.094173 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:55493: BlockingRecv error: recv got EOF from 127.28.83.67:55493 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 18:01:51 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:51 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:51.694944 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:51.689022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:35185)
0810 18:01:51.689490 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:51.689507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:51.689649 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:51.689979 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:51.689982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:51.690248 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:51.690378 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:51.690385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.691145 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.691148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:51.691698 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:51.691705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.692146 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.692149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:51.692212 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:51.692803 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:51.692815 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:51.694768 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":92,"threads_started":1}
W20260810 18:01:51.695009 5987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:51.689098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:35185 (local address 127.28.83.126:43629)
0810 18:01:51.689397 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:51.689423 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:51.689596 (+ 173us) server_negotiation.cc:407] Connection header received
0810 18:01:51.689763 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:51.689766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:51.689830 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:51.689920 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:51.690493 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.691032 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.691803 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.692020 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.692095 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:51.694888 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:35185: BlockingRecv error: recv got EOF from 127.28.83.65:35185 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
Aug 10 18:01:51 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:51 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:51.898162 5988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:51.891999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:34889)
0810 18:01:51.892351 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:51.892396 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:51.892532 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:51.892868 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:51.892870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:51.893070 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:51.893175 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:51.893181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.894005 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.894008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:51.894636 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:51.894645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.895153 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.895156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:51.895225 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:51.895886 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:51.895900 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:51.897984 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
W20260810 18:01:51.898257 5987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:51.892110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:34889 (local address 127.28.83.126:43629)
0810 18:01:51.892417 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:51.892421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:51.892462 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:01:51.892660 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:51.892662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:51.892715 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:51.892789 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:51.893292 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.893877 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.894744 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:51.895032 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:51.895112 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:51.898126 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:34889: BlockingRecv error: recv got EOF from 127.28.83.66:34889 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 18:01:52 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:52 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:52.101567 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:52.095218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:45677)
0810 18:01:52.095617 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:52.095636 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:52.095806 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:52.096157 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:52.096160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:52.096385 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:52.096488 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:52.096494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.097367 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.097372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:52.098016 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:52.098025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.098558 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.098562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:52.098661 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:52.099333 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:52.099346 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:52.101325 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":160,"threads_started":1}
W20260810 18:01:52.101701 5987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:52.095413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:45677 (local address 127.28.83.126:43629)
0810 18:01:52.095617 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:52.095623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:52.095738 (+ 115us) server_negotiation.cc:407] Connection header received
0810 18:01:52.095924 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:52.095928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:52.095990 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:52.096095 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:52.096668 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.097239 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.098169 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.098424 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.098555 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:52.101542 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:45677: BlockingRecv error: recv got EOF from 127.28.83.67:45677 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 10 18:01:52 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:52 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:52.701987 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:52.695896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:50377)
0810 18:01:52.696192 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:52.696207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:52.696298 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:52.696512 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:52.696514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:52.696701 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:52.696806 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:52.696811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.698004 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.698008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:52.698751 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:52.698763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.699270 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.699273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:52.699362 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:52.699884 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:52.699896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:52.701814 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
W20260810 18:01:52.702140 5991 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:52.695971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:50377 (local address 127.28.83.126:43629)
0810 18:01:52.696314 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:52.696318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:52.696332 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:01:52.696372 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:52.696374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:52.696415 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:52.696501 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:52.696939 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.697805 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.698889 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.699139 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.699262 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:52.701959 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:50377: BlockingRecv error: recv got EOF from 127.28.83.65:50377 (error 108)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":179,"threads_started":1}
Aug 10 18:01:52 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:52 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:52.905606 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:52.899119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:60201)
0810 18:01:52.899440 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:52.899454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:52.899544 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:52.899826 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:52.899828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:52.900030 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:52.900142 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:52.900148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.901274 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.901278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:52.902187 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:52.902200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.902676 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.902679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:52.902741 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:52.903249 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:52.903261 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:52.905472 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":125,"threads_started":1}
W20260810 18:01:52.905750 5991 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:52.899325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:60201 (local address 127.28.83.126:43629)
0810 18:01:52.899476 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:52.899480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:52.899497 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:01:52.899641 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:52.899643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:52.899687 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:52.899772 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:52.900274 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.901101 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.902318 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:52.902565 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:52.902647 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:52.905621 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:60201: BlockingRecv error: recv got EOF from 127.28.83.66:60201 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 18:01:53 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:53 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:53.108528 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:53.102593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:57425)
0810 18:01:53.102937 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:53.102950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:53.103093 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:53.103422 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:53.103425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:53.103649 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:53.103741 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:53.103747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.104591 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.104597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:53.105188 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:53.105197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.105771 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.105775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:53.105861 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:53.106412 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:53.106423 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:53.108415 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":98,"threads_started":1}
W20260810 18:01:53.108777 5991 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:53.102643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:57425 (local address 127.28.83.126:43629)
0810 18:01:53.102799 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:53.102804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:53.103032 (+ 228us) server_negotiation.cc:407] Connection header received
0810 18:01:53.103206 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:53.103209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:53.103310 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:53.103385 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:53.103866 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.104470 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.105322 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.105648 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.105765 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:53.108645 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:57425: BlockingRecv error: recv got EOF from 127.28.83.67:57425 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 18:01:53 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:53 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:53.709055 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:53.702949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:52277)
0810 18:01:53.703261 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:53.703275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:53.703408 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:53.703737 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:53.703741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:53.704051 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:53.704155 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:53.704161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.704975 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.704980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:53.705626 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:53.705636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.706117 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.706120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:53.706227 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:53.706869 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:53.706881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:53.708881 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":135,"threads_started":1}
W20260810 18:01:53.709143 5995 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:53.703046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:52277 (local address 127.28.83.126:43629)
0810 18:01:53.703317 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:53.703322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:53.703360 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:01:53.703529 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:53.703535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:53.703595 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:53.703718 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:53.704283 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.704856 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.705756 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.705989 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.706113 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:53.709013 (+ 2900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:52277: BlockingRecv error: recv got EOF from 127.28.83.65:52277 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
Aug 10 18:01:53 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:53 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:53.916687 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:53.906644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:32991)
0810 18:01:53.906937 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:53.906960 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:53.907090 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:53.907409 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:53.907411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:53.907607 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:53.907718 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:53.907723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.908559 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.908562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:53.909163 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:53.909171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.909658 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.909661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:53.909724 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:53.914092 (+ 4368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:53.914105 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:53.916486 (+ 2381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
W20260810 18:01:53.916728 5995 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:53.906682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:32991 (local address 127.28.83.126:43629)
0810 18:01:53.906832 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:53.906836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:53.907032 (+ 196us) server_negotiation.cc:407] Connection header received
0810 18:01:53.907224 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:53.907230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:53.907291 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:53.907373 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:53.907827 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.908448 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.909282 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:53.909548 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:53.909642 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:53.916624 (+ 6982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:32991: BlockingRecv error: recv got EOF from 127.28.83.66:32991 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 18:01:54 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:54 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:54.116317 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:54.109402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:35817)
0810 18:01:54.109825 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:54.109839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:54.109979 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:54.110277 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:54.110280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:54.110514 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:54.110644 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:54.110656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.111889 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.111893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:54.112845 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:54.112872 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.113486 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.113493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:54.113582 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:54.114155 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:54.114170 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:54.116134 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":86,"threads_started":1}
W20260810 18:01:54.116355 5995 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:54.109501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:35817 (local address 127.28.83.126:43629)
0810 18:01:54.109650 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:54.109657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:54.109920 (+ 263us) server_negotiation.cc:407] Connection header received
0810 18:01:54.110092 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:54.110095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:54.110155 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:54.110259 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:54.110806 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.111729 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.113021 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.113331 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.113565 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:54.116255 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:35817: BlockingRecv error: recv got EOF from 127.28.83.67:35817 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 18:01:54 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:54 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:54.715848 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:54.709925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:36561)
0810 18:01:54.710293 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:54.710306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:54.710408 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:54.710678 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:54.710681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:54.710866 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:54.710964 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:54.710969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.711775 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.711780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:54.712345 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:54.712353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.712830 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.712834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:54.712920 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:54.713524 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:54.713536 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:54.715731 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":96,"threads_started":1}
W20260810 18:01:54.716055 5999 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:54.710050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:36561 (local address 127.28.83.126:43629)
0810 18:01:54.710324 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:54.710327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:54.710361 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:01:54.710514 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:54.710516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:54.710560 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:54.710641 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:54.711116 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.711659 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.712471 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.712711 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.712824 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:54.715870 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:36561: BlockingRecv error: recv got EOF from 127.28.83.65:36561 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":68,"threads_started":1}
Aug 10 18:01:54 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:54 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:54.922791 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:54.917480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:43913)
0810 18:01:54.917787 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:54.917810 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:54.917907 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:54.918150 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:54.918152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:54.918344 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:54.918439 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:54.918444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.919134 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.919136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:54.919681 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:54.919687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.920105 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.920108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:54.920170 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:54.920739 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:54.920751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:54.922673 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
W20260810 18:01:54.922916 5999 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:54.917523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:43913 (local address 127.28.83.126:43629)
0810 18:01:54.917674 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:54.917678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:54.917879 (+ 201us) server_negotiation.cc:407] Connection header received
0810 18:01:54.917971 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:54.917973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:54.918016 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:54.918085 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:54.918539 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.919042 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.919777 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:54.919985 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:54.920058 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:54.922803 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:43913: BlockingRecv error: recv got EOF from 127.28.83.66:43913 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 18:01:55 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:55 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:55.126927 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:55.117274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:36435)
0810 18:01:55.117596 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:55.117610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:55.117717 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:55.117937 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:55.117939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:55.118115 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:55.118229 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:55.118234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.118966 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.118968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:55.119507 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:55.119514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.119947 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.119949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:55.120009 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:55.124924 (+ 4915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:55.124937 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:55.126803 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
W20260810 18:01:55.127171 5999 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:55.117354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:36435 (local address 127.28.83.126:43629)
0810 18:01:55.117504 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:55.117508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:55.117679 (+ 171us) server_negotiation.cc:407] Connection header received
0810 18:01:55.117784 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:55.117785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:55.117830 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:55.117897 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:55.118326 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.118852 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.119606 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.119833 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.119912 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:55.127025 (+ 7113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:36435: BlockingRecv error: recv got EOF from 127.28.83.67:36435 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 10 18:01:55 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:55 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:55.722468 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:55.716658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:35855)
0810 18:01:55.717052 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:55.717066 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:55.717202 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:55.717464 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:55.717466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:55.717668 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:55.717765 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:55.717771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.718700 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.718705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:55.719383 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:55.719393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.719862 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.719865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:55.719950 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:55.720454 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:55.720466 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:55.722350 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":89,"threads_started":1}
W20260810 18:01:55.722635 6003 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:55.716857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:35855 (local address 127.28.83.126:43629)
0810 18:01:55.717112 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:55.717115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:55.717146 (+ 31us) server_negotiation.cc:407] Connection header received
0810 18:01:55.717291 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:55.717293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:55.717339 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:55.717436 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:55.717926 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.718575 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.719512 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.719747 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.719855 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:55.722496 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:35855: BlockingRecv error: recv got EOF from 127.28.83.65:35855 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
Aug 10 18:01:55 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:55 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:55.929998 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:55.923640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:38813)
0810 18:01:55.923971 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:55.923993 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:55.924143 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:55.924433 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:55.924436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:55.924664 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:55.924757 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:55.924762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.925762 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.925767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:55.926434 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:55.926451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.926942 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.926946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:55.927067 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:55.927752 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:55.927765 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:55.929880 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
W20260810 18:01:55.930218 6003 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:55.923766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:38813 (local address 127.28.83.126:43629)
0810 18:01:55.923929 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:55.923934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:55.924112 (+ 178us) server_negotiation.cc:407] Connection header received
0810 18:01:55.924257 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:55.924261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:55.924321 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:55.924409 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:55.924879 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.925652 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.926566 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:55.926801 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:55.926923 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:55.930026 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:38813: BlockingRecv error: recv got EOF from 127.28.83.66:38813 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 18:01:56 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:56 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:56.133641 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:56.127658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:35633)
0810 18:01:56.127938 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:56.127951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:56.128052 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:56.128290 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:56.128292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:56.128474 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:56.128575 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:56.128580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.129280 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.129283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:56.129855 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:56.129861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.130348 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.130354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:56.130432 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:56.131069 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:56.131083 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:56.133521 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
W20260810 18:01:56.133805 6003 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:56.127749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:35633 (local address 127.28.83.126:43629)
0810 18:01:56.127892 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:56.127895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:56.128003 (+ 108us) server_negotiation.cc:407] Connection header received
0810 18:01:56.128139 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:56.128141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:56.128188 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:56.128249 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:56.128680 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.129188 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.129957 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.130227 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.130310 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:56.133646 (+ 3336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:35633: BlockingRecv error: recv got EOF from 127.28.83.67:35633 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 18:01:56 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:56 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:56.728840 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:56.723320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:39089)
0810 18:01:56.723606 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:56.723619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:56.723719 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:56.723962 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:56.723964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:56.724196 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:56.724299 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:56.724306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.725057 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.725060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:56.725646 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:56.725655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.726067 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.726069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:56.726133 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:56.726705 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:56.726716 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:56.728664 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
W20260810 18:01:56.729038 6007 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:56.723382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:39089 (local address 127.28.83.126:43629)
0810 18:01:56.723620 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:56.723623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:56.723685 (+ 62us) server_negotiation.cc:407] Connection header received
0810 18:01:56.723793 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:56.723796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:56.723839 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:56.723921 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:56.724402 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.724951 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.725745 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.725955 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.726027 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:56.728864 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:39089: BlockingRecv error: recv got EOF from 127.28.83.65:39089 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
Aug 10 18:01:56 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:56 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:56.936542 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:56.930819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:36223)
0810 18:01:56.931250 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:56.931268 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:56.931397 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:56.931711 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:56.931717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:56.931927 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:56.932027 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:56.932033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.932777 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.932780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:56.933367 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:56.933375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.933855 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.933858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:56.933920 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:56.934452 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:56.934464 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:56.936393 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":109,"threads_started":1}
W20260810 18:01:56.936666 6007 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:56.930849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:36223 (local address 127.28.83.126:43629)
0810 18:01:56.931036 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:56.931040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:56.931354 (+ 314us) server_negotiation.cc:407] Connection header received
0810 18:01:56.931496 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:56.931498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:56.931572 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:56.931659 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:56.932141 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.932672 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.933525 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:56.933753 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:56.933833 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:56.936544 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:36223: BlockingRecv error: recv got EOF from 127.28.83.66:36223 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 10 18:01:57 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:57 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:57.140180 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:57.134578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:49479)
0810 18:01:57.134859 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:57.134881 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:57.134981 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:57.135230 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:57.135232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:57.135418 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:57.135520 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:57.135525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.136250 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.136252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:57.136797 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:57.136804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.137239 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.137243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:57.137318 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:57.137923 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:57.137936 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:57.140048 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
W20260810 18:01:57.140372 6007 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:57.134674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:49479 (local address 127.28.83.126:43629)
0810 18:01:57.134809 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:57.134814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:57.134951 (+ 137us) server_negotiation.cc:407] Connection header received
0810 18:01:57.135061 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:57.135063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:57.135110 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:57.135183 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:57.135621 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.136138 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.136897 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.137108 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.137183 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:57.140196 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:49479: BlockingRecv error: recv got EOF from 127.28.83.67:49479 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 18:01:57 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:57 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:57.735688 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:57.729848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:55109)
0810 18:01:57.730210 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:57.730224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:57.730329 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:57.730578 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:57.730580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:57.730767 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:57.730864 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:57.730889 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.731718 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.731723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:57.732290 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:57.732298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.732801 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.732804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:57.732900 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:57.733571 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:57.733586 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:57.735528 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":101,"threads_started":1}
W20260810 18:01:57.735821 6011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:57.729912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:55109 (local address 127.28.83.126:43629)
0810 18:01:57.730223 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:57.730226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:57.730301 (+ 75us) server_negotiation.cc:407] Connection header received
0810 18:01:57.730409 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:57.730411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:57.730457 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:57.730536 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:57.731017 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.731593 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.732420 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.732665 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.732779 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:57.735685 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:55109: BlockingRecv error: recv got EOF from 127.28.83.65:55109 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":118,"threads_started":1}
Aug 10 18:01:57 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:57 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:57.943416 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:57.937494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:41323)
0810 18:01:57.937782 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:57.937796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:57.937893 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:57.938228 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:57.938231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:57.938429 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:57.938531 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:57.938551 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.939378 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.939382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:57.939946 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:57.939955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.940436 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.940438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:57.940500 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:57.941172 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:57.941184 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:57.943221 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":73,"threads_started":1}
W20260810 18:01:57.943498 6011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:57.937567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:41323 (local address 127.28.83.126:43629)
0810 18:01:57.937804 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:57.937807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:57.937872 (+ 65us) server_negotiation.cc:407] Connection header received
0810 18:01:57.938020 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:57.938023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:57.938075 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:57.938162 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:57.938664 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.939266 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.940055 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:57.940330 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:57.940412 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:57.943375 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:41323: BlockingRecv error: recv got EOF from 127.28.83.66:41323 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Aug 10 18:01:58 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:58 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:58.146517 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:58.141059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:60717)
0810 18:01:58.141347 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:58.141362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:58.141490 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:58.141761 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:58.141763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:58.141951 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:58.142055 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:58.142061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.142799 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.142801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:58.143382 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:58.143390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.143836 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.143839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:58.143920 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:58.144486 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:58.144497 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:58.146381 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":115,"threads_started":1}
W20260810 18:01:58.146651 6011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:58.141215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:60717 (local address 127.28.83.126:43629)
0810 18:01:58.141351 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:58.141355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:58.141432 (+ 77us) server_negotiation.cc:407] Connection header received
0810 18:01:58.141575 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:58.141578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:58.141625 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:58.141698 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:58.142175 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.142697 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.143486 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.143695 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.143779 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:58.146527 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:60717: BlockingRecv error: recv got EOF from 127.28.83.67:60717 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 18:01:58 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:58 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:58.742650 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:58.736718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:60129)
0810 18:01:58.737006 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:58.737020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:58.737120 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:58.737372 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:58.737375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:58.737589 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:58.737689 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:58.737694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.738652 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.738656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:58.739479 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:58.739489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.739930 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.739933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:58.739992 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:58.740524 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:58.740536 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:58.742453 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
W20260810 18:01:58.742733 6015 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:58.736841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:60129 (local address 127.28.83.126:43629)
0810 18:01:58.737102 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:58.737106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:58.737121 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:01:58.737193 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:58.737196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:58.737248 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:58.737335 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:58.737798 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.738526 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.739598 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.739827 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.739908 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:58.742585 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:60129: BlockingRecv error: recv got EOF from 127.28.83.65:60129 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
Aug 10 18:01:58 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:58 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:58.950248 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:58.944446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:40603)
0810 18:01:58.944730 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:58.944745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:58.944848 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:58.945121 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:58.945124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:58.945316 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:58.945452 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:58.945459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.946192 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.946195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:58.946931 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:58.946944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.947373 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.947376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:58.947437 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:58.947956 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:58.947968 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:58.950074 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
W20260810 18:01:58.950461 6015 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:58.944543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:40603 (local address 127.28.83.126:43629)
0810 18:01:58.944694 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:58.944697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:58.944817 (+ 120us) server_negotiation.cc:407] Connection header received
0810 18:01:58.944931 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:58.944933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:58.944978 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:58.945052 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:58.945562 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.946092 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.947056 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:58.947272 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:58.947350 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:58.950295 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:40603: BlockingRecv error: recv got EOF from 127.28.83.66:40603 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 18:01:59 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:59 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:59.153370 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:59.147545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:40703)
0810 18:01:59.147839 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:59.147853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:59.147956 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:59.148222 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:59.148224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:59.148413 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:59.148515 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:59.148520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.149376 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.149380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:59.150039 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:59.150049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.150524 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.150528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:59.150622 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:59.151257 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:59.151269 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:59.153190 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
W20260810 18:01:59.153466 6015 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:59.147699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:40703 (local address 127.28.83.126:43629)
0810 18:01:59.147896 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:59.147900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:59.147918 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:01:59.148035 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:59.148038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:59.148090 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:59.148171 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:59.148642 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.149253 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.150173 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.150402 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.150518 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:59.153323 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:40703: BlockingRecv error: recv got EOF from 127.28.83.67:40703 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 10 18:01:59 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:59 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:59.749482 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:59.743581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:45835)
0810 18:01:59.743939 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:59.743957 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:59.744075 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:59.744386 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:59.744388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:59.744632 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:59.744772 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:59.744780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.745741 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.745743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:59.746298 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:59.746306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.746778 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.746781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:59.746844 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:59.747420 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:59.747437 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:59.749298 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":95,"threads_started":1}
W20260810 18:01:59.749692 6019 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:59.743731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:45835 (local address 127.28.83.126:43629)
0810 18:01:59.744026 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:59.744031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:59.744063 (+ 32us) server_negotiation.cc:407] Connection header received
0810 18:01:59.744176 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:59.744181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:59.744239 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:59.744341 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:59.744903 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.745625 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.746431 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.746672 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.746765 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:59.749490 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:45835: BlockingRecv error: recv got EOF from 127.28.83.65:45835 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
Aug 10 18:01:59 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:01:59 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:01:59.958523 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:01:59.951159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:47807)
0810 18:01:59.951556 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 18:01:59.951573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:01:59.951763 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:01:59.952100 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:01:59.952103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:01:59.952407 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:01:59.952552 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:59.952578 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.954464 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.954468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:59.955145 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:01:59.955154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.955589 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.955592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:01:59.955652 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:59.956245 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:01:59.956265 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:01:59.958366 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":100,"threads_started":1}
W20260810 18:01:59.958650 6019 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:01:59.951200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:47807 (local address 127.28.83.126:43629)
0810 18:01:59.951359 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:01:59.951363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:01:59.951652 (+ 289us) server_negotiation.cc:407] Connection header received
0810 18:01:59.951894 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:01:59.951897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:01:59.951970 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:01:59.952068 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:01:59.952728 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.953597 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.955259 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:01:59.955482 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:01:59.955580 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:01:59.958523 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:47807: BlockingRecv error: recv got EOF from 127.28.83.66:47807 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 18:02:00 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:02:00 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:02:00.165920 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:00.154348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:41437)
0810 18:02:00.154661 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:00.154680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:00.154786 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:00.155177 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:00.155180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:00.155475 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:00.155631 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:00.155641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.156703 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.156706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:00.157323 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:00.157332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.157852 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.157855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:00.157925 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:00.163443 (+ 5518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:00.163460 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:00.165736 (+ 2276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":124,"threads_started":1}
W20260810 18:02:00.166026 6019 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:00.154493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:41437 (local address 127.28.83.126:43629)
0810 18:02:00.154657 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:00.154661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:00.154757 (+ 96us) server_negotiation.cc:407] Connection header received
0810 18:02:00.154891 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:00.154894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:00.154954 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:00.155053 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:00.155801 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.156573 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.157471 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.157720 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.157804 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:00.165900 (+ 8096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:41437: BlockingRecv error: recv got EOF from 127.28.83.67:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 18:02:00 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:02:00 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:02:00.756896 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:00.750450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:42401)
0810 18:02:00.750737 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:00.750760 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:00.750882 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:00.751176 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:00.751179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:00.751376 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:00.751484 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:00.751489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.752275 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.752277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:00.752836 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:00.752844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.753635 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.753639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:00.753735 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:00.754485 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:00.754508 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:00.756773 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
W20260810 18:02:00.757107 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:00.750560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:42401 (local address 127.28.83.126:43629)
0810 18:02:00.750847 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:00.750850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:00.750868 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:02:00.750975 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:00.750978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:00.751023 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:00.751121 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:00.751620 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.752167 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.752973 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.753480 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.753599 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:00.756926 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:42401: BlockingRecv error: recv got EOF from 127.28.83.65:42401 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":108,"threads_started":1}
Aug 10 18:02:00 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:02:00 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:02:00.965448 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:00.959457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:55637)
0810 18:02:00.959883 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:00.959896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:00.960028 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:00.960340 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:00.960342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:00.960569 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:00.960661 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:00.960667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.961466 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.961469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:00.962045 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:00.962054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.962529 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.962533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:00.962619 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:00.963203 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:00.963217 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:00.965218 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":128,"threads_started":1}
W20260810 18:02:00.965572 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:00.959589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:55637 (local address 127.28.83.126:43629)
0810 18:02:00.959740 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:00.959746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:00.959977 (+ 231us) server_negotiation.cc:407] Connection header received
0810 18:02:00.960141 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:00.960145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:00.960200 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:00.960314 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:00.960785 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.961315 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.962176 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:00.962406 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:00.962515 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:00.965456 (+ 2941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:55637: BlockingRecv error: recv got EOF from 127.28.83.66:55637 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 18:02:01 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:02:01 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:02:01.172560 6025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:01.166788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:34567)
0810 18:02:01.167141 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:01.167155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:01.167255 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:01.167503 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:01.167505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:01.167695 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:01.167795 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:01.167800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.168532 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.168535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:01.169156 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:01.169165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.169652 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.169655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:01.169717 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:01.170226 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:01.170237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:01.172402 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":146,"threads_started":1}
W20260810 18:02:01.172698 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:01.166877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:34567 (local address 127.28.83.126:43629)
0810 18:02:01.167044 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:01.167047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:01.167232 (+ 185us) server_negotiation.cc:407] Connection header received
0810 18:02:01.167337 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:01.167339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:01.167388 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:01.167460 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:01.167903 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.168430 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.169268 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.169555 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.169636 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:01.172574 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:34567: BlockingRecv error: recv got EOF from 127.28.83.67:34567 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 18:02:01 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:02:01 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:02:01.763345 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:01.757741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:48943)
0810 18:02:01.758042 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:01.758055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:01.758145 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:01.758373 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:01.758375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:01.758571 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:01.758669 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:01.758674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.759466 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.759469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:01.760018 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:01.760025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.760486 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.760488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:01.760557 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:01.761108 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:01.761120 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:01.763166 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":124,"threads_started":1}
W20260810 18:02:01.763507 6027 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:01.757784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:48943 (local address 127.28.83.126:43629)
0810 18:02:01.758097 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:01.758102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:01.758117 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:02:01.758232 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:01.758235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:01.758281 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:01.758363 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:01.758802 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.759359 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.760139 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.760375 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.760460 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:01.763313 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:48943: BlockingRecv error: recv got EOF from 127.28.83.65:48943 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
Aug 10 18:02:01 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:02:01 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:02:01.972365 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:01.966381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:59837)
0810 18:02:01.966699 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:01.966712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:01.966816 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:01.967134 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:01.967136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:01.967331 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:01.967432 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:01.967437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.968218 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.968221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:01.968790 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:01.968798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.969273 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.969275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:01.969338 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:01.969945 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:01.969958 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:01.972230 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":118,"threads_started":1}
W20260810 18:02:01.972532 6027 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:01.966444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:59837 (local address 127.28.83.126:43629)
0810 18:02:01.966599 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:01.966604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:01.966819 (+ 215us) server_negotiation.cc:407] Connection header received
0810 18:02:01.966930 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:01.966934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:01.966997 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:01.967090 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:01.967542 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.968109 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.968919 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:01.969160 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:01.969254 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:01.972371 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:59837: BlockingRecv error: recv got EOF from 127.28.83.66:59837 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 18:02:02 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:02:02 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.67@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:02:02.179533 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:02.173543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.67:58975)
0810 18:02:02.173903 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:02.173921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:02.174031 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:02.174369 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:02.174372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:02.174616 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:02.174775 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:02.174784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.175703 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.175706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:02.176277 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:02.176284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.176722 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.176725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:02.176785 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:02.177323 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:02.177334 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:02.179355 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
W20260810 18:02:02.179597 6027 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:02.173610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:58975 (local address 127.28.83.126:43629)
0810 18:02:02.173769 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:02.173773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:02.174009 (+ 236us) server_negotiation.cc:407] Connection header received
0810 18:02:02.174160 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:02.174167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:02.174216 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:02.174311 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:02.174959 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.175593 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.176392 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.176620 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.176703 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:02.179485 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.67:58975: BlockingRecv error: recv got EOF from 127.28.83.67:58975 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 18:02:02 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:02:02 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.65@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:02:02.770588 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:02.764680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.65:56515)
0810 18:02:02.765020 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:02.765037 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:02.765157 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:02.765465 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:02.765468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:02.765750 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:02.765848 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:02.765854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.766658 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.766663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:02.767264 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:02.767273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.767757 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.767761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:02.767850 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:02.768431 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:02.768444 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:02.770469 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
W20260810 18:02:02.770766 6031 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:02.764765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:56515 (local address 127.28.83.126:43629)
0810 18:02:02.765046 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:02.765052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:02.765101 (+ 49us) server_negotiation.cc:407] Connection header received
0810 18:02:02.765260 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:02.765263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:02.765316 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:02.765418 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:02.765975 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.766535 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.767392 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.767638 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.767749 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:02.770642 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.65:56515: BlockingRecv error: recv got EOF from 127.28.83.65:56515 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
Aug 10 18:02:02 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:02:02 dist-test-slave-srgv krb5kdc[4960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.83.66@KRBTEST.COM for oryx/127.28.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:02:02.978665 6032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:02.973284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43629 (local address 127.28.83.66:57303)
0810 18:02:02.973557 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:02.973582 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:02.973668 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:02.973927 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:02.973929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:02.974126 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:02.974223 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:02.974228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.974949 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.974951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:02.975524 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:02.975532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.975975 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.975978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:02.976040 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:02.976575 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:02.976587 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:02.978486 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.83.126:43629: Server oryx/127.28.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
W20260810 18:02:02.978847 6031 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:02.973460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:57303 (local address 127.28.83.126:43629)
0810 18:02:02.973620 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:02.973623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:02.973637 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:02:02.973757 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:02.973759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:02.973805 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:02.973884 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:02.974326 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.974849 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.975638 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:02.975870 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:02.975946 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:02.978687 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.83.66:57303: BlockingRecv error: recv got EOF from 127.28.83.66:57303 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:02:03.061481 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 5550
I20260810 18:02:03.068446 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 5704
I20260810 18:02:03.074954 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 5837
I20260810 18:02:03.081915 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 5478
2026-08-10T18:02:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18065 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T18:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:02:03Z Disabled control of system clock
I20260810 18:02:03.114554 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:35369
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:35369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 18:02:03.227249 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:03.227531 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:03.227566 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:03.231752 6036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:02:03.231820 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:03.231835 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:03.231853 6036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:02:03.231886 6036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:02:03.236799 6036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:35369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:35369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:03.237998 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:03.239423 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:03.248703 6041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:03.248744 6042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:03.248764 6036 server_base.cc:1034] running on GCE node
W20260810 18:02:03.248703 6044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:03.249727 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:03.250973 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:03.252204 6036 hybrid_clock.cc:648] HybridClock initialized: now 1786384923252183 us; error 51 us; skew 500 ppm
I20260810 18:02:03.254344 6036 webserver.cc:460] Webserver started at http://127.28.83.126:39691/ using document root <none> and password file <none>
I20260810 18:02:03.254973 6036 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:03.255075 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:03.255326 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:03.258013 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "44b0916bd69a4f48880424d2f5e5f776"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "88d1c05d94afc6353c254e1e656fb8de"
server_key_iv: "45c4183919e82ea2c19f2de3e8dd86c5"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.258736 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "44b0916bd69a4f48880424d2f5e5f776"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "88d1c05d94afc6353c254e1e656fb8de"
server_key_iv: "45c4183919e82ea2c19f2de3e8dd86c5"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.262811 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:02:03.265451 6050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.266696 6036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:02:03.266860 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "44b0916bd69a4f48880424d2f5e5f776"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "88d1c05d94afc6353c254e1e656fb8de"
server_key_iv: "45c4183919e82ea2c19f2de3e8dd86c5"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.266990 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:03.314462 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:03.315263 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:03.315495 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:03.322773 6036 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:35369
I20260810 18:02:03.322793 6104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:35369 every 8 connection(s)
I20260810 18:02:03.323736 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:02:03.326889 6105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:03.331772 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 6036
I20260810 18:02:03.331907 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:02:03.332182 29005 external_mini_cluster.cc:1428] Setting key a2fbea77be85ec1f160f64344f4592f4
I20260810 18:02:03.332473 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776: Bootstrap starting.
I20260810 18:02:03.335273 6105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:03.336177 6105 log.cc:824] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:03.338433 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776: No bootstrap required, opened a new log
I20260810 18:02:03.342991 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:03.333628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.28.83.126:35369)
0810 18:02:03.334338 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:03.334357 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:03.334400 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:02:03.335281 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:03.335305 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:03.335838 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:03.336207 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:03.336664 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.338046 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.338999 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.339692 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.339893 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:03.340719 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:03.340725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:03.340741 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:02:03.341247 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:03.341320 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:02:03.341348 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:03.341478 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:03.341616 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 18:02:03.341916 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":112,"threads_started":1}
I20260810 18:02:03.343191 6105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b0916bd69a4f48880424d2f5e5f776" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35369 } }
I20260810 18:02:03.343458 6105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:03.343546 6105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b0916bd69a4f48880424d2f5e5f776, State: Initialized, Role: FOLLOWER
I20260810 18:02:03.343973 6105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b0916bd69a4f48880424d2f5e5f776" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35369 } }
I20260810 18:02:03.344110 6105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:02:03.344247 6105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:02:03.344377 6105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:03.345518 6105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b0916bd69a4f48880424d2f5e5f776" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35369 } }
I20260810 18:02:03.345858 6105 leader_election.cc:302] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44b0916bd69a4f48880424d2f5e5f776; no voters:
I20260810 18:02:03.346175 6105 leader_election.cc:288] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:02:03.346294 6110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:02:03.346553 6110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [term 1 LEADER]: Becoming Leader. State: Replica: 44b0916bd69a4f48880424d2f5e5f776, State: Running, Role: LEADER
I20260810 18:02:03.346920 6110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b0916bd69a4f48880424d2f5e5f776" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35369 } }
I20260810 18:02:03.347393 6105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:02:03.348520 6112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44b0916bd69a4f48880424d2f5e5f776. Latest consensus state: current_term: 1 leader_uuid: "44b0916bd69a4f48880424d2f5e5f776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b0916bd69a4f48880424d2f5e5f776" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35369 } } }
I20260810 18:02:03.348672 6112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [sys.catalog]: This master's current role is: LEADER
I20260810 18:02:03.349059 6119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:02:03.349064 6111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44b0916bd69a4f48880424d2f5e5f776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b0916bd69a4f48880424d2f5e5f776" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 35369 } } }
I20260810 18:02:03.349952 6111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776 [sys.catalog]: This master's current role is: LEADER
I20260810 18:02:03.352739 6119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:02:03.358769 6119 catalog_manager.cc:1347] Generated new cluster ID: 2d747d5e70af48c8be2c901a38134600
I20260810 18:02:03.358851 6119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:02:03.391919 6119 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:02:03.392910 6119 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:02:03.400802 6119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44b0916bd69a4f48880424d2f5e5f776: Generated new TSK 0
I20260810 18:02:03.401614 6119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:02:03.407021 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35369
--builtin_ntp_servers=127.28.83.84:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 18:02:03.522233 6129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:03.522526 6129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:03.522565 6129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:03.526806 6129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:03.526871 6129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:03.526925 6129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:02:03.531778 6129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:03.532889 6129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:03.534355 6129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:03.543978 6136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:03.544030 6135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:03.543993 6138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:03.544229 6129 server_base.cc:1034] running on GCE node
I20260810 18:02:03.544868 6129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:03.545650 6129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:03.546855 6129 hybrid_clock.cc:648] HybridClock initialized: now 1786384923546837 us; error 32 us; skew 500 ppm
I20260810 18:02:03.548986 6129 webserver.cc:460] Webserver started at http://127.28.83.65:44349/ using document root <none> and password file <none>
I20260810 18:02:03.549710 6129 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:03.549810 6129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:03.550036 6129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:03.552120 6129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "3abf367822484d428864f2b116fb43b5"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "d82818db292a771d46aab52af9eda0f8"
server_key_iv: "4bf834083c016e860ad93b4258b5c032"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.552729 6129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "3abf367822484d428864f2b116fb43b5"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "d82818db292a771d46aab52af9eda0f8"
server_key_iv: "4bf834083c016e860ad93b4258b5c032"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.556453 6129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:02:03.559666 6144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.561479 6129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:02:03.561640 6129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "3abf367822484d428864f2b116fb43b5"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "d82818db292a771d46aab52af9eda0f8"
server_key_iv: "4bf834083c016e860ad93b4258b5c032"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.561862 6129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:03.575456 6129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:03.576138 6129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:03.576295 6129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:03.576840 6129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:03.577843 6129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:03.577891 6129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.577931 6129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:03.577946 6129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.586939 6129 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:34595
I20260810 18:02:03.586963 6259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:34595 every 8 connection(s)
I20260810 18:02:03.587903 6129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:02:03.594348 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 6129
I20260810 18:02:03.594445 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:02:03.594741 29005 external_mini_cluster.cc:1428] Setting key f20232f103005d376c809f00d3c78ad2
I20260810 18:02:03.596664 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35369
--builtin_ntp_servers=127.28.83.84:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 18:02:03.597646 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:03.589881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:43577 (local address 127.28.83.126:35369)
0810 18:02:03.590017 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:03.590022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:03.591007 (+ 985us) server_negotiation.cc:407] Connection header received
0810 18:02:03.592431 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:03.592434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:03.592495 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:03.592596 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:03.593531 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.594088 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.595485 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.595720 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.595833 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:03.597092 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:03.597096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:03.597099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:02:03.597112 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:03.597139 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:02:03.597147 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:03.597241 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:03.597434 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 18:02:03.597495 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:02:03.598562 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:03.590197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:35369 (local address 127.28.83.65:43577)
0810 18:02:03.590852 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:03.590892 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:03.592198 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:03.592777 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:03.592784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:03.592865 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:03.593348 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:03.593360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.594221 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.594227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:03.595361 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:03.595370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.595873 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.595877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:03.596089 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:03.596805 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:02:03.596808 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:03.596862 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:02:03.596936 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:03.597245 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:03.597256 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:03.597263 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:02:03.597398 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 18:02:03.597784 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":126,"threads_started":1}
I20260810 18:02:03.599889 6260 heartbeater.cc:347] Connected to a master server at 127.28.83.126:35369
I20260810 18:02:03.600162 6260 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:03.600677 6260 heartbeater.cc:511] Master 127.28.83.126:35369 requested a full tablet report, sending...
I20260810 18:02:03.602232 6068 ts_manager.cc:194] Registered new tserver with Master: 3abf367822484d428864f2b116fb43b5 (127.28.83.65:34595)
I20260810 18:02:03.603349 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.65:43577
W20260810 18:02:03.710322 6263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:03.710600 6263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:03.710638 6263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:03.714815 6263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:03.714879 6263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:03.714934 6263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:02:03.719834 6263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:03.721024 6263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:03.722486 6263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:03.732355 6268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:03.732414 6271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:03.732426 6269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:03.732795 6263 server_base.cc:1034] running on GCE node
I20260810 18:02:03.733266 6263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:03.734037 6263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:03.735242 6263 hybrid_clock.cc:648] HybridClock initialized: now 1786384923735211 us; error 51 us; skew 500 ppm
I20260810 18:02:03.737102 6263 webserver.cc:460] Webserver started at http://127.28.83.66:40529/ using document root <none> and password file <none>
I20260810 18:02:03.737761 6263 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:03.737856 6263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:03.738123 6263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:03.740389 6263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "6d2765b0bef94be0bc3f8db26d8acec6"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "ba205e05446bc6c09c7fc47ab5959562"
server_key_iv: "c281a9873ebc730e8f7b429146f87df6"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.740993 6263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "6d2765b0bef94be0bc3f8db26d8acec6"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "ba205e05446bc6c09c7fc47ab5959562"
server_key_iv: "c281a9873ebc730e8f7b429146f87df6"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.744798 6263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 18:02:03.747555 6277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.748952 6263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:02:03.749116 6263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "6d2765b0bef94be0bc3f8db26d8acec6"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "ba205e05446bc6c09c7fc47ab5959562"
server_key_iv: "c281a9873ebc730e8f7b429146f87df6"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.749267 6263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:03.788272 6263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:03.789052 6263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:03.789312 6263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:03.790011 6263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:03.791011 6263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:03.791095 6263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.791172 6263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:03.791227 6263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.800469 6263 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:46425
I20260810 18:02:03.800503 6394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:46425 every 8 connection(s)
I20260810 18:02:03.801461 6263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:02:03.805014 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 6263
I20260810 18:02:03.805154 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:02:03.805498 29005 external_mini_cluster.cc:1428] Setting key 900a742f6e41eceab655ee509fbfbf48
I20260810 18:02:03.807499 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35369
--builtin_ntp_servers=127.28.83.84:41717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 18:02:03.812101 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:03.803294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:48271 (local address 127.28.83.126:35369)
0810 18:02:03.803446 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:03.803451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:03.804609 (+ 1158us) server_negotiation.cc:407] Connection header received
0810 18:02:03.806368 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:03.806372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:03.806428 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:03.806538 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:03.807561 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.808173 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.809879 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.810159 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.810257 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:03.811588 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:03.811591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:03.811593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:02:03.811606 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:03.811625 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:02:03.811633 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:03.811693 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:03.811909 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 18:02:03.811986 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:02:03.813472 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:03.803792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:35369 (local address 127.28.83.66:48271)
0810 18:02:03.804457 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:03.804499 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:03.806149 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:03.806725 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:03.806732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:03.806853 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:03.807419 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:03.807433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.808306 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.808310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:03.809688 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:03.809704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.810338 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.810345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:03.810591 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:03.811265 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:02:03.811269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:03.811325 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:02:03.811416 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:03.811724 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:03.811738 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:03.811747 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:02:03.811893 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 18:02:03.812245 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260810 18:02:03.814656 6395 heartbeater.cc:347] Connected to a master server at 127.28.83.126:35369
I20260810 18:02:03.814944 6395 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:03.815490 6395 heartbeater.cc:511] Master 127.28.83.126:35369 requested a full tablet report, sending...
I20260810 18:02:03.816582 6068 ts_manager.cc:194] Registered new tserver with Master: 6d2765b0bef94be0bc3f8db26d8acec6 (127.28.83.66:46425)
I20260810 18:02:03.817155 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.66:48271
W20260810 18:02:03.922869 6398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:03.923177 6398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:03.923215 6398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:03.927449 6398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:03.927517 6398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:03.927573 6398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:02:03.932725 6398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:35369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:03.933913 6398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:03.935288 6398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:03.945044 6404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:03.945051 6406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:03.945436 6403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:03.945914 6398 server_base.cc:1034] running on GCE node
I20260810 18:02:03.946447 6398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:03.947304 6398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:03.948488 6398 hybrid_clock.cc:648] HybridClock initialized: now 1786384923948456 us; error 51 us; skew 500 ppm
I20260810 18:02:03.951308 6398 webserver.cc:460] Webserver started at http://127.28.83.67:36853/ using document root <none> and password file <none>
I20260810 18:02:03.952036 6398 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:03.952133 6398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:03.952415 6398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:03.954988 6398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "7650da8cff664d81913d19991282a25c"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "ffe3d6421e4f54d20506b5e6531b68f6"
server_key_iv: "7b366a0844e3c0be86c50e6831a738ce"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.955518 6398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "7650da8cff664d81913d19991282a25c"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "ffe3d6421e4f54d20506b5e6531b68f6"
server_key_iv: "7b366a0844e3c0be86c50e6831a738ce"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.959323 6398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:02:03.961889 6412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.963124 6398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:02:03.963248 6398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "7650da8cff664d81913d19991282a25c"
format_stamp: "Formatted at 2026-08-10 18:02:03 on dist-test-slave-srgv"
server_key: "ffe3d6421e4f54d20506b5e6531b68f6"
server_key_iv: "7b366a0844e3c0be86c50e6831a738ce"
server_key_version: "encryptionkey@0"
I20260810 18:02:03.963357 6398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:03.974334 6398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:03.975037 6398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:03.975246 6398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:03.975853 6398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:03.976817 6398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:03.976866 6398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.976907 6398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:03.976922 6398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:03.986156 6398 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:36109
I20260810 18:02:03.986183 6530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:36109 every 8 connection(s)
I20260810 18:02:03.987120 6398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:02:03.994983 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 6398
I20260810 18:02:03.995121 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:02:03.995464 29005 external_mini_cluster.cc:1428] Setting key d5c9fc6834657ef82f2c9fcc793142dc
I20260810 18:02:03.996541 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:03.988978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:38343 (local address 127.28.83.126:35369)
0810 18:02:03.989128 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:03.989134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:03.990247 (+ 1113us) server_negotiation.cc:407] Connection header received
0810 18:02:03.991563 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:03.991565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:03.991621 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:03.991700 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:03.992557 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.993096 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.994266 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.994494 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.994584 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:03.995866 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:03.995870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:03.995873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:02:03.995888 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:03.995912 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:02:03.995925 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:03.996014 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:03.996307 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 18:02:03.996381 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:02:03.997462 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:03.989394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:35369 (local address 127.28.83.67:38343)
0810 18:02:03.990072 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:03.990113 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:03.991378 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:03.991865 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:03.991871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:03.991950 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:03.992432 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:03.992444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.993226 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.993229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:03.994146 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:03.994155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:03.994669 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:03.994676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:03.994884 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:03.995552 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:02:03.995554 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:03.995607 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:02:03.995695 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:03.996026 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:03.996036 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:03.996042 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:02:03.996181 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 18:02:03.996484 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260810 18:02:03.998564 6531 heartbeater.cc:347] Connected to a master server at 127.28.83.126:35369
I20260810 18:02:03.998766 6531 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:03.999228 6531 heartbeater.cc:511] Master 127.28.83.126:35369 requested a full tablet report, sending...
I20260810 18:02:04.000295 6068 ts_manager.cc:194] Registered new tserver with Master: 7650da8cff664d81913d19991282a25c (127.28.83.67:36109)
I20260810 18:02:04.000877 6068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.67:38343
I20260810 18:02:04.009668 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:02:04.012548 6108 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:04.011620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.28.83.126:35369)
0810 18:02:04.011810 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:04.011813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:04.011871 (+ 58us) server_negotiation.cc:407] Connection header received
0810 18:02:04.011994 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:04.011996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:04.012038 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:04.012105 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:04.012417 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49364: BlockingRecv error: recv got EOF from 127.0.0.1:49364 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 18:02:04.012754 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 6129
I20260810 18:02:04.020493 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 6263
I20260810 18:02:04.027607 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 6398
I20260810 18:02:04.033560 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 6036
2026-08-10T18:02:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (950 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T18:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:02:04Z Disabled control of system clock
I20260810 18:02:04.064337 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33669
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:33669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:02:04.191993 6542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:04.192281 6542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:04.192371 6542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:04.196595 6542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:02:04.196660 6542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:04.196676 6542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:04.196722 6542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:02:04.196750 6542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:02:04.202767 6542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:33669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:04.203878 6542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:04.205327 6542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:04.213697 6547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:04.213712 6542 server_base.cc:1034] running on GCE node
W20260810 18:02:04.213830 6548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:04.213712 6550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:04.214432 6542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:04.215487 6542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:04.216673 6542 hybrid_clock.cc:648] HybridClock initialized: now 1786384924216660 us; error 25 us; skew 500 ppm
I20260810 18:02:04.218547 6542 webserver.cc:460] Webserver started at http://127.28.83.126:37909/ using document root <none> and password file <none>
I20260810 18:02:04.219115 6542 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:04.219206 6542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:04.219434 6542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:04.221556 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "fb1c833a19cf44a8a53ed8f8564c38ec"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "c171eaec412497972c860afbb6fbef10"
server_key_iv: "a91c2cc0775dc3fb130d9d24a880accd"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.222143 6542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "fb1c833a19cf44a8a53ed8f8564c38ec"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "c171eaec412497972c860afbb6fbef10"
server_key_iv: "a91c2cc0775dc3fb130d9d24a880accd"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.225901 6542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:02:04.228277 6556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.229746 6542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:02:04.229897 6542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "fb1c833a19cf44a8a53ed8f8564c38ec"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "c171eaec412497972c860afbb6fbef10"
server_key_iv: "a91c2cc0775dc3fb130d9d24a880accd"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.230014 6542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:04.234535 6542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:04.235162 6542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:04.235355 6542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:04.242599 6542 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:33669
I20260810 18:02:04.242607 6611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:33669 every 8 connection(s)
I20260810 18:02:04.243548 6542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:02:04.247000 6612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:04.250567 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 6542
I20260810 18:02:04.250695 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:02:04.250988 29005 external_mini_cluster.cc:1428] Setting key eb5bc0c66b0ebdbd06ac20d19cd1c53a
I20260810 18:02:04.253268 6612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec: Bootstrap starting.
I20260810 18:02:04.255705 6612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:04.256498 6612 log.cc:824] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:04.256816 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:04.252497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.28.83.126:33669)
0810 18:02:04.253149 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:04.253166 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:04.253200 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:02:04.253939 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:04.253958 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:04.254317 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:04.254688 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:04.255490 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:04.255494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:04.255506 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:02:04.255517 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:04.255572 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:02:04.255587 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:04.255711 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:04.255815 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 18:02:04.256090 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260810 18:02:04.258769 6612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec: No bootstrap required, opened a new log
I20260810 18:02:04.262485 6612 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1c833a19cf44a8a53ed8f8564c38ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33669 } }
I20260810 18:02:04.262794 6612 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:04.262878 6612 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1c833a19cf44a8a53ed8f8564c38ec, State: Initialized, Role: FOLLOWER
I20260810 18:02:04.263343 6612 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1c833a19cf44a8a53ed8f8564c38ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33669 } }
I20260810 18:02:04.263492 6612 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:02:04.263556 6612 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:02:04.263660 6612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:04.264667 6612 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1c833a19cf44a8a53ed8f8564c38ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33669 } }
I20260810 18:02:04.265009 6612 leader_election.cc:302] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb1c833a19cf44a8a53ed8f8564c38ec; no voters:
I20260810 18:02:04.265338 6612 leader_election.cc:288] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:02:04.265504 6617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:02:04.265753 6617 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [term 1 LEADER]: Becoming Leader. State: Replica: fb1c833a19cf44a8a53ed8f8564c38ec, State: Running, Role: LEADER
I20260810 18:02:04.266072 6617 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1c833a19cf44a8a53ed8f8564c38ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33669 } }
I20260810 18:02:04.266654 6612 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:02:04.267769 6618 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb1c833a19cf44a8a53ed8f8564c38ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1c833a19cf44a8a53ed8f8564c38ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33669 } } }
I20260810 18:02:04.267895 6618 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [sys.catalog]: This master's current role is: LEADER
I20260810 18:02:04.268093 6619 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb1c833a19cf44a8a53ed8f8564c38ec. Latest consensus state: current_term: 1 leader_uuid: "fb1c833a19cf44a8a53ed8f8564c38ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1c833a19cf44a8a53ed8f8564c38ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33669 } } }
I20260810 18:02:04.268203 6619 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec [sys.catalog]: This master's current role is: LEADER
I20260810 18:02:04.268471 6626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:02:04.271194 6626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:02:04.276525 6626 catalog_manager.cc:1347] Generated new cluster ID: 8e9ecad4eb764d3a9de9e8fb13dc253e
I20260810 18:02:04.276605 6626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:02:04.285820 6626 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:02:04.286191 6626 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:02:04.294008 6626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb1c833a19cf44a8a53ed8f8564c38ec: Generated new TSK 0
I20260810 18:02:04.294847 6626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:02:04.300729 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33669
--builtin_ntp_servers=127.28.83.84:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:02:04.422029 6638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:04.422308 6638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:04.422350 6638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:04.426343 6638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:04.426406 6638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:04.426462 6638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:02:04.431145 6638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:04.432225 6638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:04.433853 6638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:04.444099 6644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:04.444238 6638 server_base.cc:1034] running on GCE node
W20260810 18:02:04.444092 6643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:04.444466 6647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:04.444942 6638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:04.445662 6638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:04.446856 6638 hybrid_clock.cc:648] HybridClock initialized: now 1786384924446834 us; error 30 us; skew 500 ppm
I20260810 18:02:04.448971 6638 webserver.cc:460] Webserver started at http://127.28.83.65:33235/ using document root <none> and password file <none>
I20260810 18:02:04.449672 6638 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:04.449743 6638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:04.450105 6638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:04.454654 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "5faf6aaa004540b79e911e4456531cdb"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "94ff29c1d25b72f68795365cbf048b13"
server_key_iv: "41838085a1b6c6e08189514b8ca87d01"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.455323 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "5faf6aaa004540b79e911e4456531cdb"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "94ff29c1d25b72f68795365cbf048b13"
server_key_iv: "41838085a1b6c6e08189514b8ca87d01"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.459442 6638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:02:04.462095 6653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.463320 6638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:02:04.463464 6638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "5faf6aaa004540b79e911e4456531cdb"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "94ff29c1d25b72f68795365cbf048b13"
server_key_iv: "41838085a1b6c6e08189514b8ca87d01"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.463589 6638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:04.468472 6638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:04.469133 6638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:04.469360 6638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:04.469944 6638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:04.470884 6638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:04.470958 6638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.471032 6638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:04.471086 6638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.480084 6638 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:34987
I20260810 18:02:04.480154 6770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:34987 every 8 connection(s)
I20260810 18:02:04.481048 6638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:02:04.487427 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:04.482918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:36499 (local address 127.28.83.126:33669)
0810 18:02:04.483067 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:04.483071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:04.484250 (+ 1179us) server_negotiation.cc:407] Connection header received
0810 18:02:04.485618 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:04.485621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:04.485659 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:04.485737 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:04.486959 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:04.486962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:04.486965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:02:04.486978 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:04.486999 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:02:04.487009 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:04.487151 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:04.487270 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 18:02:04.487319 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:02:04.488258 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:04.483342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33669 (local address 127.28.83.65:36499)
0810 18:02:04.484096 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:04.484139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:04.485350 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:04.485903 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:04.485909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:04.485991 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:04.486697 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:02:04.486699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:04.486739 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:02:04.486810 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:04.487139 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:04.487146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:04.487150 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:02:04.487275 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 18:02:04.487459 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":130,"threads_started":1}
I20260810 18:02:04.489658 6771 heartbeater.cc:347] Connected to a master server at 127.28.83.126:33669
I20260810 18:02:04.489908 6771 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:04.490078 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 6638
I20260810 18:02:04.490227 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:02:04.490442 6771 heartbeater.cc:511] Master 127.28.83.126:33669 requested a full tablet report, sending...
I20260810 18:02:04.490564 29005 external_mini_cluster.cc:1428] Setting key bed503ebf87158dcadbf1c76952ea139
I20260810 18:02:04.492163 6573 ts_manager.cc:194] Registered new tserver with Master: 5faf6aaa004540b79e911e4456531cdb (127.28.83.65:34987)
I20260810 18:02:04.492586 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33669
--builtin_ntp_servers=127.28.83.84:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:02:04.605751 6774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:04.606037 6774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:04.606078 6774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:04.610208 6774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:04.610319 6774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:04.610422 6774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:02:04.615206 6774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:04.616248 6774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:04.617728 6774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:04.627499 6779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:04.627655 6780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:04.627539 6782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:04.627800 6774 server_base.cc:1034] running on GCE node
I20260810 18:02:04.628314 6774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:04.628974 6774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:04.630162 6774 hybrid_clock.cc:648] HybridClock initialized: now 1786384924630148 us; error 25 us; skew 500 ppm
I20260810 18:02:04.631925 6774 webserver.cc:460] Webserver started at http://127.28.83.66:44711/ using document root <none> and password file <none>
I20260810 18:02:04.632529 6774 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:04.632634 6774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:04.632872 6774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:04.635504 6774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "c69d602e28dd407c968b073ccb95303a"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "1c5829add4acadcc7ca7a565c505c22d"
server_key_iv: "d5a83e1ffbf64ace05ff9c872a0c367b"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.636099 6774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "c69d602e28dd407c968b073ccb95303a"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "1c5829add4acadcc7ca7a565c505c22d"
server_key_iv: "d5a83e1ffbf64ace05ff9c872a0c367b"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.640048 6774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:02:04.642828 6788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.644088 6774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:02:04.644237 6774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "c69d602e28dd407c968b073ccb95303a"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "1c5829add4acadcc7ca7a565c505c22d"
server_key_iv: "d5a83e1ffbf64ace05ff9c872a0c367b"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.644373 6774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:04.649374 6774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:04.650087 6774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:04.650291 6774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:04.650876 6774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:04.651841 6774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:04.651924 6774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.651993 6774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:04.652045 6774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.660966 6774 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:43395
I20260810 18:02:04.660995 6903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:43395 every 8 connection(s)
I20260810 18:02:04.661978 6774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:02:04.668429 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:04.663795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:40163 (local address 127.28.83.126:33669)
0810 18:02:04.663944 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:04.663947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:04.665020 (+ 1073us) server_negotiation.cc:407] Connection header received
0810 18:02:04.666460 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:04.666464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:04.666511 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:04.666623 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:04.667937 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:04.667940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:04.667942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:02:04.667953 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:04.667970 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:02:04.667978 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:04.668056 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:04.668228 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 18:02:04.668280 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:02:04.669381 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:04.664202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33669 (local address 127.28.83.66:40163)
0810 18:02:04.664858 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:04.664900 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:04.666231 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:04.666810 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:04.666837 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:04.666925 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:04.667561 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:02:04.667562 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:04.667626 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:02:04.667755 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:04.668057 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:04.668064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:04.668068 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:02:04.668236 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 18:02:04.668446 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260810 18:02:04.669575 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 6774
I20260810 18:02:04.669704 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:02:04.670047 29005 external_mini_cluster.cc:1428] Setting key 36720387fe8687e6568d8f4fef2fe807
I20260810 18:02:04.670589 6904 heartbeater.cc:347] Connected to a master server at 127.28.83.126:33669
I20260810 18:02:04.670837 6904 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:04.671265 6904 heartbeater.cc:511] Master 127.28.83.126:33669 requested a full tablet report, sending...
I20260810 18:02:04.672341 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33669
--builtin_ntp_servers=127.28.83.84:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 18:02:04.672432 6573 ts_manager.cc:194] Registered new tserver with Master: c69d602e28dd407c968b073ccb95303a (127.28.83.66:43395)
W20260810 18:02:04.785871 6907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:04.786159 6907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:04.786201 6907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:04.790354 6907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:04.790417 6907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:04.790516 6907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:02:04.795771 6907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:04.796932 6907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:04.798553 6907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:04.808426 6913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:04.808375 6912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:04.808375 6915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:04.808409 6907 server_base.cc:1034] running on GCE node
I20260810 18:02:04.809118 6907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:04.809804 6907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:04.810969 6907 hybrid_clock.cc:648] HybridClock initialized: now 1786384924810958 us; error 30 us; skew 500 ppm
I20260810 18:02:04.812814 6907 webserver.cc:460] Webserver started at http://127.28.83.67:44055/ using document root <none> and password file <none>
I20260810 18:02:04.813396 6907 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:04.813555 6907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:04.813796 6907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:04.815894 6907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "2b5fa98d425b40fe8412a9ce9fd15189"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "476cbafc3ad86d2c9ee61443ace273eb"
server_key_iv: "6f8979da934f5018cb6da2b713994fb0"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.816478 6907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "2b5fa98d425b40fe8412a9ce9fd15189"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "476cbafc3ad86d2c9ee61443ace273eb"
server_key_iv: "6f8979da934f5018cb6da2b713994fb0"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.820488 6907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 18:02:04.823123 6921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.824481 6907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:02:04.824622 6907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "2b5fa98d425b40fe8412a9ce9fd15189"
format_stamp: "Formatted at 2026-08-10 18:02:04 on dist-test-slave-srgv"
server_key: "476cbafc3ad86d2c9ee61443ace273eb"
server_key_iv: "6f8979da934f5018cb6da2b713994fb0"
server_key_version: "encryptionkey@0"
I20260810 18:02:04.824752 6907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:04.829615 6907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:04.830253 6907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:04.830439 6907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:04.830986 6907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:04.831933 6907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:04.832005 6907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.832072 6907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:04.832121 6907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:04.841185 6907 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:44595
I20260810 18:02:04.841205 7039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:44595 every 8 connection(s)
I20260810 18:02:04.842192 6907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:02:04.848453 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:04.844072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:43945 (local address 127.28.83.126:33669)
0810 18:02:04.844223 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:04.844227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:04.845317 (+ 1090us) server_negotiation.cc:407] Connection header received
0810 18:02:04.846679 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:04.846681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:04.846724 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:04.846809 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:04.848047 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:04.848050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:04.848053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:02:04.848065 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:04.848085 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:02:04.848094 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:04.848177 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:04.848301 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 18:02:04.848343 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:02:04.849391 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 6907
I20260810 18:02:04.849575 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:02:04.849548 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:04.844480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33669 (local address 127.28.83.67:43945)
0810 18:02:04.845144 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:04.845187 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:04.846467 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:04.846987 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:04.846993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:04.847084 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:04.847764 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:02:04.847766 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:04.847821 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:02:04.847894 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:04.848175 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:04.848182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:04.848187 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:02:04.848332 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 18:02:04.848535 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
I20260810 18:02:04.849885 29005 external_mini_cluster.cc:1428] Setting key 6d4690d610f24706b4cc3e6986c859c1
I20260810 18:02:04.850908 7040 heartbeater.cc:347] Connected to a master server at 127.28.83.126:33669
I20260810 18:02:04.851186 7040 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:04.851644 7040 heartbeater.cc:511] Master 127.28.83.126:33669 requested a full tablet report, sending...
I20260810 18:02:04.852713 6573 ts_manager.cc:194] Registered new tserver with Master: 2b5fa98d425b40fe8412a9ce9fd15189 (127.28.83.67:44595)
I20260810 18:02:04.863974 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:02:04.867012 6615 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:04.866125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.28.83.126:33669)
0810 18:02:04.866306 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:04.866309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:04.866393 (+ 84us) server_negotiation.cc:407] Connection header received
0810 18:02:04.866500 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:04.866502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:04.866538 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:04.866603 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:04.866880 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37722: BlockingRecv error: recv got EOF from 127.0.0.1:37722 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 18:02:04.867241 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 6638
I20260810 18:02:04.873929 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 6774
I20260810 18:02:04.881171 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 6907
I20260810 18:02:04.887293 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 6542
2026-08-10T18:02:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (854 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:02:04 dist-test-slave-srgv krb5kdc[7050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:02:04 dist-test-slave-srgv krb5kdc[7050](info): set up 2 sockets
Aug 10 18:02:04 dist-test-slave-srgv krb5kdc[7050](info): commencing operation
krb5kdc: starting...
W20260810 18:02:05.937398 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:02:06 dist-test-slave-srgv krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:02:06.072834 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:46293
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:46293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:02:06.187004 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:06.187352 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:06.187438 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:06.191682 7063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:02:06.191807 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:06.191849 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:06.191888 7063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:02:06.191927 7063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:02:06.197572 7063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:46293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:46293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:06.198846 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:06.199930 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:06.208837 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:06.208839 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:06.208856 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:06.209095 7063 server_base.cc:1034] running on GCE node
I20260810 18:02:06.209924 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:06.211056 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:06.212275 7063 hybrid_clock.cc:648] HybridClock initialized: now 1786384926212265 us; error 28 us; skew 500 ppm
Aug 10 18:02:06 dist-test-slave-srgv krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:02:06.220716 7063 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:02:06.221820 7063 webserver.cc:460] Webserver started at http://127.28.83.126:37267/ using document root <none> and password file <none>
I20260810 18:02:06.222409 7063 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:06.222503 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:06.222745 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:06.224869 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "edc5a15d58c346eeb086c9537466743a"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "706d40fb1095a1058ab4b2720021cbfd"
server_key_iv: "9b5f997516fd510b92738e05325369cc"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.225528 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "edc5a15d58c346eeb086c9537466743a"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "706d40fb1095a1058ab4b2720021cbfd"
server_key_iv: "9b5f997516fd510b92738e05325369cc"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.229364 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:02:06.231909 7078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:06.233114 7063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:02:06.233261 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "edc5a15d58c346eeb086c9537466743a"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "706d40fb1095a1058ab4b2720021cbfd"
server_key_iv: "9b5f997516fd510b92738e05325369cc"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.233381 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:06.250429 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:06.253690 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:06.253918 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:06.261952 7063 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:46293
I20260810 18:02:06.261986 7131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:46293 every 8 connection(s)
I20260810 18:02:06.263015 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:02:06.265918 7132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:06.270550 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 7063
I20260810 18:02:06.270709 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:02:06.270978 29005 external_mini_cluster.cc:1428] Setting key 5a476ad13abf8b2fa09e98582a0be1d7
I20260810 18:02:06.271883 7132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a: Bootstrap starting.
I20260810 18:02:06.274645 7132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:06.275530 7132 log.cc:824] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:06.277801 7132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a: No bootstrap required, opened a new log
Aug 10 18:02:06 dist-test-slave-srgv krb5kdc[7050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:02:06.282248 7132 raft_consensus.cc:348] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc5a15d58c346eeb086c9537466743a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46293 } }
I20260810 18:02:06.282562 7132 raft_consensus.cc:374] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:06.282658 7132 raft_consensus.cc:729] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc5a15d58c346eeb086c9537466743a, State: Initialized, Role: FOLLOWER
I20260810 18:02:06.283195 7132 consensus_queue.cc:260] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc5a15d58c346eeb086c9537466743a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46293 } }
I20260810 18:02:06.283368 7132 raft_consensus.cc:388] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:02:06.283442 7132 raft_consensus.cc:482] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:02:06.283550 7132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:06.284718 7132 raft_consensus.cc:504] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc5a15d58c346eeb086c9537466743a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46293 } }
I20260810 18:02:06.285120 7132 leader_election.cc:302] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edc5a15d58c346eeb086c9537466743a; no voters:
I20260810 18:02:06.285508 7132 leader_election.cc:288] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:02:06.285622 7137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:02:06.285866 7137 raft_consensus.cc:686] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [term 1 LEADER]: Becoming Leader. State: Replica: edc5a15d58c346eeb086c9537466743a, State: Running, Role: LEADER
I20260810 18:02:06.286218 7137 consensus_queue.cc:237] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc5a15d58c346eeb086c9537466743a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46293 } }
I20260810 18:02:06.286890 7132 sys_catalog.cc:564] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:02:06.287209 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:06.272493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.28.83.126:46293)
0810 18:02:06.272965 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:06.272986 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:06.273036 (+ 50us) server_negotiation.cc:407] Connection header received
0810 18:02:06.273895 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:06.273919 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:06.274362 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:06.274744 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:06.275521 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.276879 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.277828 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.278506 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.278667 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:06.281462 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:06.281469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:06.281488 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:06.281535 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:06.283754 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:06.284337 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:06.284346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:06.284355 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:06.284513 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:06.284793 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:06.284797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:06.284799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:06.285363 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:06.285596 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:06.285920 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 18:02:06.286224 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":138,"threads_started":1}
I20260810 18:02:06.287477 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [sys.catalog]: SysCatalogTable state changed. Reason: New leader edc5a15d58c346eeb086c9537466743a. Latest consensus state: current_term: 1 leader_uuid: "edc5a15d58c346eeb086c9537466743a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc5a15d58c346eeb086c9537466743a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46293 } } }
I20260810 18:02:06.287586 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [sys.catalog]: This master's current role is: LEADER
I20260810 18:02:06.288210 7139 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edc5a15d58c346eeb086c9537466743a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc5a15d58c346eeb086c9537466743a" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46293 } } }
I20260810 18:02:06.288324 7139 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a [sys.catalog]: This master's current role is: LEADER
W20260810 18:02:06.291790 7152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:02:06.291859 7152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:02:06.291942 7143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:02:06.292646 7143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:02:06.298671 7143 catalog_manager.cc:1347] Generated new cluster ID: ba94fa9fd8f345d09b0d482d3545861c
I20260810 18:02:06.298748 7143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:02:06.316080 7143 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:02:06.317363 7143 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:02:06.330669 7143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a: Generated new TSK 0
I20260810 18:02:06.331483 7143 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:02:06.373092 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46293
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:02:06.493574 7158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:06.493858 7158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:06.493902 7158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:06.497956 7158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:06.498023 7158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:06.498122 7158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:02:06.503505 7158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:06.504673 7158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:06.505761 7158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:06.515822 7166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:06.515815 7163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:06.515867 7164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:06.516148 7158 server_base.cc:1034] running on GCE node
I20260810 18:02:06.516541 7158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:06.517144 7158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:06.518376 7158 hybrid_clock.cc:648] HybridClock initialized: now 1786384926518337 us; error 51 us; skew 500 ppm
Aug 10 18:02:06 dist-test-slave-srgv krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:02:06.521445 7158 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:02:06.522378 7158 webserver.cc:460] Webserver started at http://127.28.83.65:33501/ using document root <none> and password file <none>
I20260810 18:02:06.522986 7158 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:06.523090 7158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:06.523331 7158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:06.525492 7158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "5d08cd5863bc4b5db8cea83630447bd5"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "bb551595c0c5e55e1e05beaca87dc7e8"
server_key_iv: "45b1154ea7e7ac5d8bbc029c140e9214"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.526103 7158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "5d08cd5863bc4b5db8cea83630447bd5"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "bb551595c0c5e55e1e05beaca87dc7e8"
server_key_iv: "45b1154ea7e7ac5d8bbc029c140e9214"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.529883 7158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:02:06.532578 7173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:06.533936 7158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:02:06.534087 7158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "5d08cd5863bc4b5db8cea83630447bd5"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "bb551595c0c5e55e1e05beaca87dc7e8"
server_key_iv: "45b1154ea7e7ac5d8bbc029c140e9214"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.534217 7158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:06.559199 7158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:06.562481 7158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:06.562714 7158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:06.563366 7158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:06.564251 7158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:06.564327 7158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:06.564399 7158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:06.564450 7158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:06.573822 7158 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:39471
I20260810 18:02:06.573848 7287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:39471 every 8 connection(s)
I20260810 18:02:06.574792 7158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:02:06.581135 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 7158
I20260810 18:02:06.581260 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:02:06.581610 29005 external_mini_cluster.cc:1428] Setting key 917f3fbfeaefcf74342f94868257edc2
Aug 10 18:02:06 dist-test-slave-srgv krb5kdc[7050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:02:06.590143 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:06.576766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:33389 (local address 127.28.83.126:46293)
0810 18:02:06.576933 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:06.576937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:06.577834 (+ 897us) server_negotiation.cc:407] Connection header received
0810 18:02:06.579259 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:06.579263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:06.579321 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:06.579433 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:06.580649 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.581454 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.582844 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.583087 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.583201 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:06.585885 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:06.585889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:06.585891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:06.585916 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:06.587563 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:06.588141 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:06.588146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:06.588148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:06.588239 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:06.588687 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:06.588692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:06.588694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:06.588891 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:06.589000 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:06.589695 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 18:02:06.589876 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:02:06.590916 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:06.577161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:46293 (local address 127.28.83.65:33389)
0810 18:02:06.577678 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:06.577723 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:06.579039 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:06.579614 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:06.579621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:06.580014 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:06.580493 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:06.580506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.581601 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.581605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:06.582694 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:06.582705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.583258 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.583263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:06.583432 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:06.584084 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:06.584101 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:06.585702 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:06.587693 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:06.587699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:06.587716 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:06.588006 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:06.588335 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:06.588338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:06.588340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:06.588589 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:06.588998 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:06.589005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:06.589353 (+ 348us) client_negotiation.cc:769] Sending connection context
0810 18:02:06.589601 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 18:02:06.589957 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:02:06.592291 7288 heartbeater.cc:347] Connected to a master server at 127.28.83.126:46293
I20260810 18:02:06.592535 7288 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:06.593058 7288 heartbeater.cc:511] Master 127.28.83.126:46293 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:02:06.594903 7095 ts_manager.cc:194] Registered new tserver with Master: 5d08cd5863bc4b5db8cea83630447bd5 (127.28.83.65:39471)
I20260810 18:02:06.596527 7095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:33389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:02:06.612602 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46293
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:02:06.729277 7293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:06.729571 7293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:06.729653 7293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:06.733554 7293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:06.733647 7293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:06.733737 7293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:02:06.739284 7293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:06.740386 7293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:06.741477 7293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:06.751271 7302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:06.751283 7299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:06.751436 7293 server_base.cc:1034] running on GCE node
W20260810 18:02:06.751294 7300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:06.752031 7293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:06.752678 7293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:06.753859 7293 hybrid_clock.cc:648] HybridClock initialized: now 1786384926753831 us; error 48 us; skew 500 ppm
Aug 10 18:02:06 dist-test-slave-srgv krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:02:06.756937 7293 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:02:06.757930 7293 webserver.cc:460] Webserver started at http://127.28.83.66:44545/ using document root <none> and password file <none>
I20260810 18:02:06.758518 7293 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:06.758611 7293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:06.758852 7293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:06.760881 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "7c977f9d3b5e494692260cf36ad813f6"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "56648722e43821ac96230b27f99577a3"
server_key_iv: "e88296f4174a78694e93056a49d9491c"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.761524 7293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "7c977f9d3b5e494692260cf36ad813f6"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "56648722e43821ac96230b27f99577a3"
server_key_iv: "e88296f4174a78694e93056a49d9491c"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.765231 7293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:02:06.767766 7309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:06.769234 7293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:02:06.769346 7293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "7c977f9d3b5e494692260cf36ad813f6"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "56648722e43821ac96230b27f99577a3"
server_key_iv: "e88296f4174a78694e93056a49d9491c"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.769506 7293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:06.785197 7293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:06.788961 7293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:06.789203 7293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:06.789880 7293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:06.790704 7293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:06.790776 7293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:06.790844 7293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:06.790892 7293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:06.799800 7293 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:32805
I20260810 18:02:06.799823 7423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:32805 every 8 connection(s)
I20260810 18:02:06.800686 7293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:02:06.809705 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 7293
I20260810 18:02:06.809841 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:02:06.810102 29005 external_mini_cluster.cc:1428] Setting key 7c4ead08ce120b86bc09210dd3bf5d89
Aug 10 18:02:06 dist-test-slave-srgv krb5kdc[7050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:02:06.815023 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:06.802581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:56709 (local address 127.28.83.126:46293)
0810 18:02:06.802725 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:06.802729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:06.803528 (+ 799us) server_negotiation.cc:407] Connection header received
0810 18:02:06.805008 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:06.805011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:06.805075 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:06.805191 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:06.806488 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.807028 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.808195 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.808422 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.808534 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:06.811293 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:06.811297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:06.811300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:06.811333 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:06.812807 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:06.813340 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:06.813344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:06.813346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:06.813510 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:06.813889 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:06.813894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:06.813896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:06.814088 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:06.814205 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:06.814796 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 18:02:06.814905 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:02:06.816274 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:06.802932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:46293 (local address 127.28.83.66:56709)
0810 18:02:06.803378 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:06.803420 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:06.804795 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:06.805367 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:06.805374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:06.805805 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:06.806337 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:06.806351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.807171 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.807176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:06.808058 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:06.808067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:06.808579 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:06.808583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:06.808769 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:06.809326 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:06.809341 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:06.811123 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:06.812928 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:06.812933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:06.812949 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:06.813231 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:06.813622 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:06.813625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:06.813627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:06.813774 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:06.814204 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:06.814210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:06.814510 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 18:02:06.814737 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 18:02:06.815113 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260810 18:02:06.817487 7424 heartbeater.cc:347] Connected to a master server at 127.28.83.126:46293
I20260810 18:02:06.817778 7424 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:06.818392 7424 heartbeater.cc:511] Master 127.28.83.126:46293 requested a full tablet report, sending...
I20260810 18:02:06.819615 7095 ts_manager.cc:194] Registered new tserver with Master: 7c977f9d3b5e494692260cf36ad813f6 (127.28.83.66:32805)
I20260810 18:02:06.820209 7095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:56709
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:02:06.838826 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46293
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:02:06.953879 7429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:06.954195 7429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:06.954293 7429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:06.958575 7429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:06.958669 7429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:06.958761 7429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:02:06.964203 7429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:06.965401 7429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:06.966502 7429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:06.976440 7434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:06.976440 7435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:06.976464 7437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:06.976799 7429 server_base.cc:1034] running on GCE node
I20260810 18:02:06.977241 7429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:06.977972 7429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:06.979166 7429 hybrid_clock.cc:648] HybridClock initialized: now 1786384926979105 us; error 82 us; skew 500 ppm
Aug 10 18:02:06 dist-test-slave-srgv krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:02:06.982220 7429 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:02:06.983150 7429 webserver.cc:460] Webserver started at http://127.28.83.67:36913/ using document root <none> and password file <none>
I20260810 18:02:06.983731 7429 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:06.983825 7429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:06.984086 7429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:02:06.986289 7429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "4c606be43b9b41308f7f7f21f2939aba"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "2b65b0104d38e5f6e82ede8478e1b441"
server_key_iv: "b1cbd0db2a320b4c17d180113a278907"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.986959 7429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "4c606be43b9b41308f7f7f21f2939aba"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "2b65b0104d38e5f6e82ede8478e1b441"
server_key_iv: "b1cbd0db2a320b4c17d180113a278907"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.991408 7429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:02:06.994131 7444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:06.995483 7429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:02:06.995652 7429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "4c606be43b9b41308f7f7f21f2939aba"
format_stamp: "Formatted at 2026-08-10 18:02:06 on dist-test-slave-srgv"
server_key: "2b65b0104d38e5f6e82ede8478e1b441"
server_key_iv: "b1cbd0db2a320b4c17d180113a278907"
server_key_version: "encryptionkey@0"
I20260810 18:02:06.995785 7429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:07.012239 7429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:07.015897 7429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:07.016112 7429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:07.016777 7429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:07.017707 7429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:07.017778 7429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:07.017853 7429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:07.017902 7429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:07.026742 7429 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:37495
I20260810 18:02:07.026767 7558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:37495 every 8 connection(s)
I20260810 18:02:07.027634 7429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:02:07.036067 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 7429
I20260810 18:02:07.036247 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:02:07.036545 29005 external_mini_cluster.cc:1428] Setting key 014f9a3a6712cfdcc204f4ae52cb9e6b
Aug 10 18:02:07 dist-test-slave-srgv krb5kdc[7050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:02:07.044986 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.029493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:39731 (local address 127.28.83.126:46293)
0810 18:02:07.029675 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:07.029680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:07.030555 (+ 875us) server_negotiation.cc:407] Connection header received
0810 18:02:07.031866 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:07.031868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:07.031917 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:07.032051 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:07.033161 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.033795 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.034988 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.035230 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.035352 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.038485 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:07.038488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:07.038490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:07.038516 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:07.040076 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:07.040589 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:07.040591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:07.040593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:07.040745 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:07.041170 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:07.041173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:07.041174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:07.043606 (+ 2432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:07.043712 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:07.044710 (+ 998us) server_negotiation.cc:299] Negotiation successful
0810 18:02:07.044848 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:02:07.046051 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.029935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:46293 (local address 127.28.83.67:39731)
0810 18:02:07.030402 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:07.030444 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:07.031674 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:07.032191 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:07.032197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:07.032559 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:07.033024 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.033037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.033945 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.033955 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.034850 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.034860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.035398 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.035402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.035598 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.036344 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:07.036373 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:07.038305 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:07.040189 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:07.040195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:07.040209 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:07.040493 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:07.040866 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:07.040868 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:07.040870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:07.041022 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:07.043843 (+ 2821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:07.043854 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:07.044312 (+ 458us) client_negotiation.cc:769] Sending connection context
0810 18:02:07.044657 (+ 345us) client_negotiation.cc:240] Negotiation successful
0810 18:02:07.045022 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260810 18:02:07.047246 7559 heartbeater.cc:347] Connected to a master server at 127.28.83.126:46293
I20260810 18:02:07.047516 7559 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:07.048070 7559 heartbeater.cc:511] Master 127.28.83.126:46293 requested a full tablet report, sending...
I20260810 18:02:07.049196 7095 ts_manager.cc:194] Registered new tserver with Master: 4c606be43b9b41308f7f7f21f2939aba (127.28.83.67:37495)
I20260810 18:02:07.049935 7095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:39731
I20260810 18:02:07.051424 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:02:07.062219 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.053616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39836 (local address 127.28.83.126:46293)
0810 18:02:07.053773 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:07.053777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:07.053942 (+ 165us) server_negotiation.cc:407] Connection header received
0810 18:02:07.054095 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:07.054098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:07.054146 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:07.054244 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:07.054966 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.055643 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.056796 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.057048 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.057163 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.058768 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:07.058771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:07.058774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:07.058814 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:07.060402 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:07.060842 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:07.060846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:07.060847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:07.060931 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:07.061231 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:07.061234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:07.061236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:07.061551 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:07.061699 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:07.061934 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 18:02:07.062052 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:02:07.065092 7095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:02:07.068069 7095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:02:07.083722 7571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.077159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:37495 (local address 127.0.0.1:35874)
0810 18:02:07.077616 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:07.077675 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:07.077812 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:07.078653 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:07.078659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:07.078684 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:07.078859 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.078870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.080524 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.080528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.082255 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.082265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.083134 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.083138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.083246 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.083283 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 18:02:07.083400 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 18:02:07.083532 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":388,"thread_start_us":129,"threads_started":1}
I20260810 18:02:07.083724 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.077228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:32805 (local address 127.0.0.1:47678)
0810 18:02:07.078259 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:07.078274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:07.078367 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:07.078965 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:07.078967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:07.078985 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:07.079067 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.079075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.080615 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.080623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.082278 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.082290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.083417 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.083420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.083501 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.083521 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:02:07.083581 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 18:02:07.083638 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":211,"thread_start_us":100,"threads_started":1}
I20260810 18:02:07.083909 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.078070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.28.83.66:32805)
0810 18:02:07.078501 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:07.078513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:07.078537 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:02:07.078617 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:07.078623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:07.078801 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:07.079234 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:07.079393 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.080464 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.082444 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.083304 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.083421 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.083557 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:07.083617 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 18:02:07.083712 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":117,"threads_started":1}
I20260810 18:02:07.084275 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.077372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.28.83.67:37495)
0810 18:02:07.078123 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:07.078138 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:07.078165 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:02:07.078250 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:07.078255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:07.078467 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:07.079025 (+ 558us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:07.079187 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.080377 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.082388 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.082991 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.083132 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.083302 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:07.083967 (+ 665us) server_negotiation.cc:299] Negotiation successful
0810 18:02:07.084112 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":168,"threads_started":1}
I20260810 18:02:07.085083 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.078002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:39471 (local address 127.0.0.1:47020)
0810 18:02:07.079081 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:07.079091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:07.079207 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:07.079595 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:07.079598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:07.079616 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:07.079716 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.079724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.080706 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.080709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.082278 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.082290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.083780 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.083784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.084733 (+ 949us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.084768 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 18:02:07.084856 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 18:02:07.084939 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":102,"thread_start_us":128,"threads_started":1}
I20260810 18:02:07.085168 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.078553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.28.83.65:39471)
0810 18:02:07.078940 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:07.078957 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:07.079185 (+ 228us) server_negotiation.cc:407] Connection header received
0810 18:02:07.079322 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:07.079326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:07.079463 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:07.079613 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:07.079814 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.080576 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.082444 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.083304 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.083421 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.083557 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:07.084903 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0810 18:02:07.085009 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":87,"threads_started":1}
I20260810 18:02:07.087661 7358 tablet_service.cc:1467] Processing CreateTablet for tablet 17304d1e54354138a1e31eb27bc17bfe (DEFAULT_TABLE table=test-table [id=cc0f0be216d042b79154561eec3877c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:02:07.087663 7493 tablet_service.cc:1467] Processing CreateTablet for tablet 17304d1e54354138a1e31eb27bc17bfe (DEFAULT_TABLE table=test-table [id=cc0f0be216d042b79154561eec3877c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:02:07.087662 7222 tablet_service.cc:1467] Processing CreateTablet for tablet 17304d1e54354138a1e31eb27bc17bfe (DEFAULT_TABLE table=test-table [id=cc0f0be216d042b79154561eec3877c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:02:07.088766 7222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17304d1e54354138a1e31eb27bc17bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:07.088770 7493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17304d1e54354138a1e31eb27bc17bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:07.088771 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17304d1e54354138a1e31eb27bc17bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:07.095436 7577 tablet_bootstrap.cc:492] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: Bootstrap starting.
I20260810 18:02:07.097760 7579 tablet_bootstrap.cc:492] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: Bootstrap starting.
I20260810 18:02:07.098179 7577 tablet_bootstrap.cc:654] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:07.098981 7577 log.cc:824] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:07.099762 7579 tablet_bootstrap.cc:654] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:07.100128 7578 tablet_bootstrap.cc:492] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5: Bootstrap starting.
I20260810 18:02:07.100962 7579 log.cc:824] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:07.101581 7577 tablet_bootstrap.cc:492] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: No bootstrap required, opened a new log
I20260810 18:02:07.101796 7577 ts_tablet_manager.cc:1397] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:02:07.102391 7578 tablet_bootstrap.cc:654] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:07.102833 7579 tablet_bootstrap.cc:492] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: No bootstrap required, opened a new log
I20260810 18:02:07.103025 7579 ts_tablet_manager.cc:1397] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:02:07.103240 7578 log.cc:824] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:07.105345 7578 tablet_bootstrap.cc:492] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5: No bootstrap required, opened a new log
I20260810 18:02:07.105561 7578 ts_tablet_manager.cc:1397] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:02:07.105857 7577 raft_consensus.cc:348] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } }
I20260810 18:02:07.106150 7577 raft_consensus.cc:374] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:07.106210 7577 raft_consensus.cc:729] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c977f9d3b5e494692260cf36ad813f6, State: Initialized, Role: FOLLOWER
I20260810 18:02:07.106201 7579 raft_consensus.cc:348] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } }
I20260810 18:02:07.106518 7579 raft_consensus.cc:374] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:07.106576 7579 raft_consensus.cc:729] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c606be43b9b41308f7f7f21f2939aba, State: Initialized, Role: FOLLOWER
I20260810 18:02:07.106689 7577 consensus_queue.cc:260] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } }
I20260810 18:02:07.106964 7579 consensus_queue.cc:260] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } }
I20260810 18:02:07.107795 7577 ts_tablet_manager.cc:1428] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:02:07.107812 7579 ts_tablet_manager.cc:1428] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:02:07.108188 7424 heartbeater.cc:503] Master 127.28.83.126:46293 was elected leader, sending a full tablet report...
I20260810 18:02:07.109107 7559 heartbeater.cc:503] Master 127.28.83.126:46293 was elected leader, sending a full tablet report...
I20260810 18:02:07.110001 7578 raft_consensus.cc:348] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } }
I20260810 18:02:07.110265 7578 raft_consensus.cc:374] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:07.110332 7578 raft_consensus.cc:729] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d08cd5863bc4b5db8cea83630447bd5, State: Initialized, Role: FOLLOWER
I20260810 18:02:07.110759 7578 consensus_queue.cc:260] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } }
I20260810 18:02:07.111511 7288 heartbeater.cc:503] Master 127.28.83.126:46293 was elected leader, sending a full tablet report...
I20260810 18:02:07.111816 7578 ts_tablet_manager.cc:1428] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:02:07.161367 7585 raft_consensus.cc:482] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:02:07.161598 7585 raft_consensus.cc:504] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } }
I20260810 18:02:07.162761 7585 leader_election.cc:288] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c606be43b9b41308f7f7f21f2939aba (127.28.83.67:37495), 7c977f9d3b5e494692260cf36ad813f6 (127.28.83.66:32805)
I20260810 18:02:07.166951 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.163227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:42127 (local address 127.28.83.66:32805)
0810 18:02:07.163376 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:07.163387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:07.163668 (+ 281us) server_negotiation.cc:407] Connection header received
0810 18:02:07.163826 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:07.163830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:07.163882 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:07.163971 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:07.164267 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.164754 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.165966 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.166368 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.166504 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.166556 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:07.166747 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 18:02:07.166814 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:02:07.166971 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.163113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:48653 (local address 127.28.83.67:37495)
0810 18:02:07.163268 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:07.163275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:07.163549 (+ 274us) server_negotiation.cc:407] Connection header received
0810 18:02:07.163736 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:07.163739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:07.163792 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:07.163882 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:07.164216 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.164728 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.165966 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.166389 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.166525 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.166583 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:07.166775 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 18:02:07.166849 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:02:07.166993 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.163080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:32805 (local address 127.28.83.65:42127)
0810 18:02:07.163577 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:07.163589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:07.163718 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:07.163982 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:07.163984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:07.164000 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:07.164148 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.164154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.164866 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.164870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.165842 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.165851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.166497 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.166500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.166610 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.166636 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:02:07.166728 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:02:07.166801 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":44,"thread_start_us":123,"threads_started":1}
I20260810 18:02:07.167004 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.162989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:37495 (local address 127.28.83.65:48653)
0810 18:02:07.163437 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:07.163450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:07.163603 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:07.163895 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:07.163898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:07.163913 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:07.164078 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.164085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.164845 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.164849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.165839 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:07.165848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.166517 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.166521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:07.166641 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.166668 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:02:07.166759 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 18:02:07.166829 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":25,"thread_start_us":145,"threads_started":1}
I20260810 18:02:07.167752 7513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17304d1e54354138a1e31eb27bc17bfe" candidate_uuid: "5d08cd5863bc4b5db8cea83630447bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c606be43b9b41308f7f7f21f2939aba" is_pre_election: true
I20260810 18:02:07.167824 7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17304d1e54354138a1e31eb27bc17bfe" candidate_uuid: "5d08cd5863bc4b5db8cea83630447bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c977f9d3b5e494692260cf36ad813f6" is_pre_election: true
I20260810 18:02:07.168072 7513 raft_consensus.cc:2454] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d08cd5863bc4b5db8cea83630447bd5 in term 0.
I20260810 18:02:07.168118 7378 raft_consensus.cc:2454] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d08cd5863bc4b5db8cea83630447bd5 in term 0.
I20260810 18:02:07.168560 7176 leader_election.cc:302] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c606be43b9b41308f7f7f21f2939aba, 5d08cd5863bc4b5db8cea83630447bd5; no voters:
I20260810 18:02:07.168864 7585 raft_consensus.cc:2781] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:02:07.168952 7585 raft_consensus.cc:482] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:02:07.168995 7585 raft_consensus.cc:3037] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:07.170065 7585 raft_consensus.cc:504] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } }
I20260810 18:02:07.170437 7585 leader_election.cc:288] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [CANDIDATE]: Term 1 election: Requested vote from peers 4c606be43b9b41308f7f7f21f2939aba (127.28.83.67:37495), 7c977f9d3b5e494692260cf36ad813f6 (127.28.83.66:32805)
I20260810 18:02:07.170782 7513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17304d1e54354138a1e31eb27bc17bfe" candidate_uuid: "5d08cd5863bc4b5db8cea83630447bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c606be43b9b41308f7f7f21f2939aba"
I20260810 18:02:07.170830 7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17304d1e54354138a1e31eb27bc17bfe" candidate_uuid: "5d08cd5863bc4b5db8cea83630447bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c977f9d3b5e494692260cf36ad813f6"
I20260810 18:02:07.170897 7513 raft_consensus.cc:3037] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:07.171031 7378 raft_consensus.cc:3037] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:07.171990 7513 raft_consensus.cc:2454] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d08cd5863bc4b5db8cea83630447bd5 in term 1.
I20260810 18:02:07.172158 7378 raft_consensus.cc:2454] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d08cd5863bc4b5db8cea83630447bd5 in term 1.
I20260810 18:02:07.172405 7176 leader_election.cc:302] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c606be43b9b41308f7f7f21f2939aba, 5d08cd5863bc4b5db8cea83630447bd5; no voters:
I20260810 18:02:07.172636 7585 raft_consensus.cc:2781] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:02:07.172894 7585 raft_consensus.cc:686] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [term 1 LEADER]: Becoming Leader. State: Replica: 5d08cd5863bc4b5db8cea83630447bd5, State: Running, Role: LEADER
I20260810 18:02:07.173260 7585 consensus_queue.cc:237] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } }
I20260810 18:02:07.176548 7095 catalog_manager.cc:5495] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d08cd5863bc4b5db8cea83630447bd5 (127.28.83.65). New cstate: current_term: 1 leader_uuid: "5d08cd5863bc4b5db8cea83630447bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d08cd5863bc4b5db8cea83630447bd5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 39471 } health_report { overall_health: HEALTHY } } }
I20260810 18:02:07.207069 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.203432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.28.83.65:39471)
0810 18:02:07.203588 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:07.203593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:07.203606 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:02:07.203689 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:07.203691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:07.203753 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:07.203854 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:02:07.204112 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.204666 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.205558 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:07.205786 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:07.205874 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:07.206201 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:02:07.206719 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:02:07.206821 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:07.206896 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 18:02:07.206952 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:02:07.213529 7378 raft_consensus.cc:1261] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [term 1 FOLLOWER]: Refusing update from remote peer 5d08cd5863bc4b5db8cea83630447bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:02:07.213529 7513 raft_consensus.cc:1261] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [term 1 FOLLOWER]: Refusing update from remote peer 5d08cd5863bc4b5db8cea83630447bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:02:07.214387 7585 consensus_queue.cc:1035] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c606be43b9b41308f7f7f21f2939aba" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:02:07.214632 7588 consensus_queue.cc:1035] T 17304d1e54354138a1e31eb27bc17bfe P 5d08cd5863bc4b5db8cea83630447bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c977f9d3b5e494692260cf36ad813f6" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:02:07.222810 7592 mvcc.cc:205] Tried to move back new op lower bound from 7317032661857910784 to 7317032661705043968. Current Snapshot: MvccSnapshot[applied={T|T < 7317032661857910784}]
I20260810 18:02:07.230201 7094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:02:07.230387 7094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:02:07.232568 7094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edc5a15d58c346eeb086c9537466743a: Sending DeleteTablet for 3 replicas of tablet 17304d1e54354138a1e31eb27bc17bfe
I20260810 18:02:07.233502 7493 tablet_service.cc:1514] Processing DeleteTablet for tablet 17304d1e54354138a1e31eb27bc17bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:02:07 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:35874
I20260810 18:02:07.233604 7358 tablet_service.cc:1514] Processing DeleteTablet for tablet 17304d1e54354138a1e31eb27bc17bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:02:07 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:47678
I20260810 18:02:07.234048 7600 tablet_replica.cc:330] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: stopping tablet replica
I20260810 18:02:07.234123 7602 tablet_replica.cc:330] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: stopping tablet replica
I20260810 18:02:07.234149 7222 tablet_service.cc:1514] Processing DeleteTablet for tablet 17304d1e54354138a1e31eb27bc17bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:02:07 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:47020
I20260810 18:02:07.234395 7600 raft_consensus.cc:2229] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:02:07.234422 7602 raft_consensus.cc:2229] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:02:07.234695 7600 raft_consensus.cc:2258] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:02:07.234704 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 7158
I20260810 18:02:07.234711 7602 raft_consensus.cc:2258] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:02:07.237258 7600 ts_tablet_manager.cc:1905] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:02:07.238776 7602 ts_tablet_manager.cc:1905] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:02:07.240813 7600 ts_tablet_manager.cc:1918] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:02:07.240933 7600 log.cc:1196] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/17304d1e54354138a1e31eb27bc17bfe
I20260810 18:02:07.240981 7602 ts_tablet_manager.cc:1918] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:02:07.241087 7602 log.cc:1196] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/17304d1e54354138a1e31eb27bc17bfe
I20260810 18:02:07.241310 7600 ts_tablet_manager.cc:1939] T 17304d1e54354138a1e31eb27bc17bfe P 4c606be43b9b41308f7f7f21f2939aba: Deleting consensus metadata
I20260810 18:02:07.241503 7602 ts_tablet_manager.cc:1939] T 17304d1e54354138a1e31eb27bc17bfe P 7c977f9d3b5e494692260cf36ad813f6: Deleting consensus metadata
I20260810 18:02:07.242614 7080 catalog_manager.cc:4841] TS 7c977f9d3b5e494692260cf36ad813f6 (127.28.83.66:32805): tablet 17304d1e54354138a1e31eb27bc17bfe (table test-table [id=cc0f0be216d042b79154561eec3877c0]) successfully deleted
I20260810 18:02:07.242697 7081 catalog_manager.cc:4841] TS 4c606be43b9b41308f7f7f21f2939aba (127.28.83.67:37495): tablet 17304d1e54354138a1e31eb27bc17bfe (table test-table [id=cc0f0be216d042b79154561eec3877c0]) successfully deleted
W20260810 18:02:07.244649 7081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:39471 (error 108)
I20260810 18:02:07.244690 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 7293
I20260810 18:02:07.247359 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:07.247030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:39471 (local address 127.0.0.1:47028)
0810 18:02:07.247209 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:07.247283 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:39471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 18:02:07.247516 7081 catalog_manager.cc:4584] TS 5d08cd5863bc4b5db8cea83630447bd5 (127.28.83.65:39471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17304d1e54354138a1e31eb27bc17bfe: Network error: Client connection negotiation failed: client connection to 127.28.83.65:39471: connect: Connection refused (error 111)
I20260810 18:02:07.251608 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 7429
I20260810 18:02:07.258008 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 7063
2026-08-10T18:02:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2371 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:02:07 dist-test-slave-srgv krb5kdc[7607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:02:07 dist-test-slave-srgv krb5kdc[7607](info): set up 2 sockets
Aug 10 18:02:07 dist-test-slave-srgv krb5kdc[7607](info): commencing operation
krb5kdc: starting...
W20260810 18:02:08.317199 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:02:08 dist-test-slave-srgv krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:02:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/rangeradmin_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/rangeradmin_127.28.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/rangerlookup_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/rangerlookup_127.28.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/HTTP_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/HTTP_127.28.83.84@KRBTEST.COM.keytab.
I20260810 18:02:08.496990 29005 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5EWyHF/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/postgres -l logfile start
2026-08-10 18:02:11.063 UTC [7637] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 18:02:11.063 UTC [7637] LOG: listening on IPv4 address "127.28.83.84", port 46903
2026-08-10 18:02:11.067 UTC [7637] LOG: listening on Unix socket "/tmp/.s.PGSQL.46903"
2026-08-10 18:02:11.089 UTC [7640] LOG: database system was shut down at 2026-08-10 18:02:09 UTC
2026-08-10 18:02:11.092 UTC [7637] LOG: database system is ready to accept connections
I20260810 18:02:12.061094 29005 mini_postgres.cc:96] Postgres bound to 46903
2026-08-10 18:02:12.068 UTC [7649] FATAL: database "slave" does not exist
127.28.83.84:46903 - accepting connections
I20260810 18:02:12.069849 29005 mini_ranger.cc:161] Starting Ranger...
I20260810 18:02:12.086149 29005 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 18:02:12.579167 29005 mini_ranger.cc:87] Created ranger Postgres database
I20260810 18:02:12.579298 29005 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-admin
2026-08-10 18:02:12,906 [I] DB FLAVOR :POSTGRES
2026-08-10 18:02:12,907 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:02:12,907 [I] Checking connection..
2026-08-10 18:02:12,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 18:02:13,160 [I] Checking connection passed.
2026-08-10 18:02:13,161 [I] --------- Verifying version history table ---------
2026-08-10 18:02:13,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:02:13,427 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:02:13,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:02:13,684 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:02:13,684 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 18:02:13,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 18:02:13.945 UTC [7714] WARNING: there is no transaction in progress
2026-08-10 18:02:13,959 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 18:02:13,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:02:14,348 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 18:02:14,348 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:02:14,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 18:02:14,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 18:02:15,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-srgv.c.gcp-upstream.internal','N') ;"
2026-08-10 18:02:15,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:02:16,045 [I] Table x_portal_user does not exist in database ranger
2026-08-10 18:02:16,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:02:16,451 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 18:02:16,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:02:16,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:02:16,975 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 18:02:16,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 18:02:17.943 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:17.961 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:17.977 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:17.993 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.013 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.030 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.086 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.093 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.108 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.118 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.128 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.142 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.152 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.159 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.166 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18.173 UTC [7823] WARNING: there is no transaction in progress
2026-08-10 18:02:18,820 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 18:02:18,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 18:02:19,082 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 18:02:19,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:02:19,333 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 18:02:19,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:02:19,581 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 18:02:19,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:02:19,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:02:20,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-srgv.c.gcp-upstream.internal';"
2026-08-10 18:02:20,308 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 18:02:20,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 18:02:20,568 [I] DB_PATCHES have already been applied
I20260810 18:02:20.574419 29005 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-admin:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5EWyHF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 18:02:20.574488 29005 mini_ranger.cc:193] Using host: 127.28.83.84
I20260810 18:02:20.578676 29005 mini_ranger.cc:227] Ranger admin URL: http://127.28.83.84:42071
Aug 10, 2026 6:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:02:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:02:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:02:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:02:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:02:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 18:02:21.717453 29005 mini_ranger.cc:160] Time spent starting Ranger: real 9.648s user 0.001s sys 0.011s
Aug 10, 2026 6:02:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:02:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 6:02:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:02:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:02:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 18:02:46.014616 29005 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/rangerkms_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/rangerkms_127.28.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.83.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.83.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.83.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/HTTP_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.83.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/HTTP_127.28.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 18:02:46 dist-test-slave-srgv krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384966, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 18:02:46.126765 29005 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 18:02:46.138734 29005 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 18:02:46.992018 29005 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 18:02:46.992125 29005 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-kms
I20260810 18:02:46.992235 29005 mini_ranger_kms.cc:227] Using postgres at 127.28.83.84:46903
2026-08-10 18:02:47,390 [I] DB FLAVOR :POSTGRES
2026-08-10 18:02:47,390 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:02:47,390 [I] Checking connection
2026-08-10 18:02:47,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 18:02:47,703 [I] connection success
2026-08-10 18:02:47,703 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 18:02:47,703 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 18:02:47,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 18:02:47,986 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 18:02:47,987 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:02:47,987 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 18:02:47,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:46903/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 18:02:48,245 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 18:02:48.708911 29005 mini_ranger_kms.cc:310] Created kms service
I20260810 18:02:48.888870 29005 mini_ranger_kms.cc:326] Created kudu user
I20260810 18:02:48.940829 29005 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 18:02:48 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384966, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:02:49.375424 29005 mini_ranger_kms.cc:384] Added ranger policy
I20260810 18:02:49.375624 29005 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-kms:/tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 18:02:49.375654 29005 mini_ranger_kms.cc:240] Using host: 127.28.83.84
I20260810 18:02:49.379944 29005 mini_ranger_kms.cc:276] Ranger KMS PID: 8005
I20260810 18:02:49.380092 29005 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.83.84:57469
Aug 10, 2026 6:02:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:02:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:02:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:02:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:02:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:02:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 18:02:51.676191 29005 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.549s user 0.016s sys 0.362s
I20260810 18:02:51.676329 29005 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 18:02:51.676434 29005 mini_ranger_kms.cc:401] 127.28.83.84:57469/kms/v1/keys
Aug 10 18:02:51 dist-test-slave-srgv krb5kdc[7607](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786384971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.83.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 18:02:52 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.83.84@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
Aug 10 18:02:52 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.83.84@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
Aug 10 18:02:55 dist-test-slave-srgv krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:02:55.492381 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33573
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:33573
--ranger_config_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:57469/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:02:55.656648 8055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:55.656934 8055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:55.656975 8055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:55.661316 8055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:02:55.661387 8055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:55.661550 8055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:55.661641 8055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:02:55.661661 8055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:02:55.667687 8055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.83.84:57469/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:33573
--ranger_config_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:55.668758 8055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:55.669863 8055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:55.679538 8061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:55.679508 8060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:55.679574 8055 server_base.cc:1034] running on GCE node
W20260810 18:02:55.679482 8063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:55.680372 8055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:55.681679 8055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:55.682838 8055 hybrid_clock.cc:648] HybridClock initialized: now 1786384975682804 us; error 47 us; skew 500 ppm
Aug 10 18:02:55 dist-test-slave-srgv krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384975, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:02:55.685874 8055 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:02:55.687004 8055 webserver.cc:460] Webserver started at http://127.28.83.126:43927/ using document root <none> and password file <none>
I20260810 18:02:55.687565 8055 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:55.687657 8055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:55.687883 8055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:02:55 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384975, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:02:55.754958 8055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "01dc61bd58264c3289fc6cdfd6cf0a57"
format_stamp: "Formatted at 2026-08-10 18:02:55 on dist-test-slave-srgv"
server_key: "23284b507e64737dde67162a3a464e60"
server_key_iv: "47643a1035a0b6a1f8e8f184d6de8521"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:55.755815 8055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "01dc61bd58264c3289fc6cdfd6cf0a57"
format_stamp: "Formatted at 2026-08-10 18:02:55 on dist-test-slave-srgv"
server_key: "23284b507e64737dde67162a3a464e60"
server_key_iv: "47643a1035a0b6a1f8e8f184d6de8521"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:55.761003 8055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 18:02:55.779866 8070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:55.781705 8055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:02:55.781952 8055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "01dc61bd58264c3289fc6cdfd6cf0a57"
format_stamp: "Formatted at 2026-08-10 18:02:55 on dist-test-slave-srgv"
server_key: "23284b507e64737dde67162a3a464e60"
server_key_iv: "47643a1035a0b6a1f8e8f184d6de8521"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:55.782106 8055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:55.819399 8055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:55.823141 8055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:55.823413 8055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:55.837944 8055 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:33573
I20260810 18:02:55.838557 8123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:33573 every 8 connection(s)
I20260810 18:02:55.839321 8055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:02:55.843219 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 8055
I20260810 18:02:55.843367 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:02:55.843712 29005 external_mini_cluster.cc:1428] Setting key 0902617a544e5957f44d3c00106c644a
I20260810 18:02:55.843806 8124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:55.852078 8124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57: Bootstrap starting.
Aug 10 18:02:55 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:02:55.855538 8124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:55.856873 8124 log.cc:824] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:55.859640 8124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57: No bootstrap required, opened a new log
I20260810 18:02:55.862664 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:55.845377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.28.83.126:33573)
0810 18:02:55.846584 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:55.846638 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:55.846686 (+ 48us) server_negotiation.cc:407] Connection header received
0810 18:02:55.848044 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:55.848069 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:55.848502 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:55.848900 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:55.849878 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:55.851058 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:55.852279 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:55.852752 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:55.852872 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:55.856159 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:55.856167 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:55.856184 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:55.856217 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:55.859124 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:55.859584 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:55.859590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:55.859596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:55.859742 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:55.860064 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:55.860081 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:55.860083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:55.860666 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:55.860907 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:55.861135 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 18:02:55.861452 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973,"thread_start_us":188,"threads_started":1}
I20260810 18:02:55.864800 8124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dc61bd58264c3289fc6cdfd6cf0a57" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33573 } }
I20260810 18:02:55.865075 8124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:55.865293 8124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01dc61bd58264c3289fc6cdfd6cf0a57, State: Initialized, Role: FOLLOWER
I20260810 18:02:55.865762 8124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dc61bd58264c3289fc6cdfd6cf0a57" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33573 } }
I20260810 18:02:55.865924 8124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:02:55.866015 8124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:02:55.866137 8124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:55.867446 8124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dc61bd58264c3289fc6cdfd6cf0a57" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33573 } }
I20260810 18:02:55.867807 8124 leader_election.cc:302] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01dc61bd58264c3289fc6cdfd6cf0a57; no voters:
I20260810 18:02:55.868132 8124 leader_election.cc:288] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:02:55.868285 8129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:02:55.868527 8129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [term 1 LEADER]: Becoming Leader. State: Replica: 01dc61bd58264c3289fc6cdfd6cf0a57, State: Running, Role: LEADER
I20260810 18:02:55.868896 8129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dc61bd58264c3289fc6cdfd6cf0a57" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33573 } }
I20260810 18:02:55.869398 8124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:02:55.870867 8131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01dc61bd58264c3289fc6cdfd6cf0a57. Latest consensus state: current_term: 1 leader_uuid: "01dc61bd58264c3289fc6cdfd6cf0a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dc61bd58264c3289fc6cdfd6cf0a57" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33573 } } }
I20260810 18:02:55.871021 8131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [sys.catalog]: This master's current role is: LEADER
I20260810 18:02:55.870893 8130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01dc61bd58264c3289fc6cdfd6cf0a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dc61bd58264c3289fc6cdfd6cf0a57" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33573 } } }
I20260810 18:02:55.871327 8130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57 [sys.catalog]: This master's current role is: LEADER
I20260810 18:02:55.871711 8138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:02:55.875738 8138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:02:55.876278 8124 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 18:02:55.888298 8138 catalog_manager.cc:1347] Generated new cluster ID: cb15a2cc972e4e58bd087a8d99c7969c
I20260810 18:02:55.888618 8138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:02:55.898641 8138 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:02:55.900185 8138 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:02:55.915469 8138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57: Generated new TSK 0
I20260810 18:02:55.916813 8138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:02:56.885531 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:57469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33573
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:02:57.075871 8169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:57.076170 8169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:57.076262 8169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:57.081038 8169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:57.081126 8169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:57.081229 8169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:02:57.087260 8169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.83.84:57469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:57.088596 8169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:57.089794 8169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:57.100477 8179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:57.100880 8177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:57.100524 8176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:57.102470 8169 server_base.cc:1034] running on GCE node
I20260810 18:02:57.103004 8169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:57.104128 8169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:57.105340 8169 hybrid_clock.cc:648] HybridClock initialized: now 1786384977105291 us; error 62 us; skew 500 ppm
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:02:57.109006 8169 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:02:57.110191 8169 webserver.cc:460] Webserver started at http://127.28.83.65:36043/ using document root <none> and password file <none>
I20260810 18:02:57.110821 8169 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:57.110921 8169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:57.111184 8169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:02:57.126902 8169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "400899edd8b244139df5ebb50fadd680"
format_stamp: "Formatted at 2026-08-10 18:02:57 on dist-test-slave-srgv"
server_key: "b9b0108bbd349ed00466a918dd51276f"
server_key_iv: "8df1c36230270da9c779ba1c7e1cbdfb"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:57.127990 8169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "400899edd8b244139df5ebb50fadd680"
format_stamp: "Formatted at 2026-08-10 18:02:57 on dist-test-slave-srgv"
server_key: "b9b0108bbd349ed00466a918dd51276f"
server_key_iv: "8df1c36230270da9c779ba1c7e1cbdfb"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:57.132962 8169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:02:57.151559 8186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:57.153256 8169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 18:02:57.153389 8169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "400899edd8b244139df5ebb50fadd680"
format_stamp: "Formatted at 2026-08-10 18:02:57 on dist-test-slave-srgv"
server_key: "b9b0108bbd349ed00466a918dd51276f"
server_key_iv: "8df1c36230270da9c779ba1c7e1cbdfb"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:57.153543 8169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:57.190822 8169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:57.195503 8169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:57.195744 8169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:57.196545 8169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:57.197659 8169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:57.197772 8169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:57.197871 8169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:57.197917 8169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:57.234539 8169 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:42045
I20260810 18:02:57.236289 8169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:02:57.236775 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 8169
I20260810 18:02:57.236920 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:02:57.237294 29005 external_mini_cluster.cc:1428] Setting key 939a3aa1971eb4fa2e4c8332f77b0d45
I20260810 18:02:57.242561 8302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:42045 every 8 connection(s)
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:02:57.263659 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:57.245353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:44601 (local address 127.28.83.126:33573)
0810 18:02:57.246036 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:57.246059 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:57.246855 (+ 796us) server_negotiation.cc:407] Connection header received
0810 18:02:57.248154 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:57.248157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:57.248219 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:57.248327 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:57.250521 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:57.251364 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:57.253320 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:57.253781 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:57.253901 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:57.257591 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:57.257595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:57.257598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:57.257629 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:57.260622 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:57.261361 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:57.261364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:57.261366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:57.261488 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:57.262002 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:57.262006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:57.262008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:57.262323 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:57.262422 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:57.263308 (+ 886us) server_negotiation.cc:299] Negotiation successful
0810 18:02:57.263456 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":135,"threads_started":1}
I20260810 18:02:57.264809 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:57.245967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33573 (local address 127.28.83.65:44601)
0810 18:02:57.246661 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:57.246717 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:57.247919 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:57.248682 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:57.248691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:57.249327 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:57.250312 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:57.250336 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:57.251548 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:57.251552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:57.253157 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:57.253169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:57.254009 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:57.254014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:57.254297 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:57.255134 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:57.255155 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:57.257300 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:57.260788 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:57.260794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:57.260813 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:57.261225 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:57.261619 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:57.261622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:57.261624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:57.261872 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:57.262505 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:57.262512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:57.262883 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 18:02:57.263200 (+ 317us) client_negotiation.cc:240] Negotiation successful
0810 18:02:57.263571 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":235,"threads_started":1}
I20260810 18:02:57.267035 8303 heartbeater.cc:347] Connected to a master server at 127.28.83.126:33573
I20260810 18:02:57.267309 8303 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:57.268258 8303 heartbeater.cc:511] Master 127.28.83.126:33573 requested a full tablet report, sending...
I20260810 18:02:57.270407 8088 ts_manager.cc:194] Registered new tserver with Master: 400899edd8b244139df5ebb50fadd680 (127.28.83.65:42045)
I20260810 18:02:57.272297 8088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:44601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:02:57.279045 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:57469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33573
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 18:02:57.466951 8309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:57.467213 8309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:57.467257 8309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:57.471625 8309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:57.471693 8309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:57.471783 8309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:02:57.478340 8309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.83.84:57469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:57.479552 8309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:57.480580 8309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:57.494436 8316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:57.494339 8315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:57.494879 8318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:57.495692 8309 server_base.cc:1034] running on GCE node
I20260810 18:02:57.496181 8309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:57.496968 8309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:57.498134 8309 hybrid_clock.cc:648] HybridClock initialized: now 1786384977498126 us; error 50 us; skew 500 ppm
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:02:57.502408 8309 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:02:57.503582 8309 webserver.cc:460] Webserver started at http://127.28.83.66:43319/ using document root <none> and password file <none>
I20260810 18:02:57.504482 8309 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:57.504611 8309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:57.504881 8309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:02:57.531133 8309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "f949591780ea41f385dac86fb239c2ff"
format_stamp: "Formatted at 2026-08-10 18:02:57 on dist-test-slave-srgv"
server_key: "fa2e827e84593d66b8f8f7445c578999"
server_key_iv: "0dcc35a06e3e0593425c7476f7f21958"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:57.532016 8309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "f949591780ea41f385dac86fb239c2ff"
format_stamp: "Formatted at 2026-08-10 18:02:57 on dist-test-slave-srgv"
server_key: "fa2e827e84593d66b8f8f7445c578999"
server_key_iv: "0dcc35a06e3e0593425c7476f7f21958"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:57.537205 8309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 18:02:57.567205 8325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:57.569123 8309 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 18:02:57.569281 8309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "f949591780ea41f385dac86fb239c2ff"
format_stamp: "Formatted at 2026-08-10 18:02:57 on dist-test-slave-srgv"
server_key: "fa2e827e84593d66b8f8f7445c578999"
server_key_iv: "0dcc35a06e3e0593425c7476f7f21958"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:57.569525 8309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:57.604904 8309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:57.608494 8309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:57.608826 8309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:57.609911 8309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:57.610988 8309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:57.611071 8309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:57.611182 8309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:57.611228 8309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:57.634119 8309 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:37125
I20260810 18:02:57.634440 8439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:37125 every 8 connection(s)
I20260810 18:02:57.635344 8309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:02:57.635660 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 8309
I20260810 18:02:57.635823 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:02:57.636209 29005 external_mini_cluster.cc:1428] Setting key d004a854ae73174c92d2dd6e767da3b3
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
I20260810 18:02:57.655845 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:57.642867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:49399 (local address 127.28.83.126:33573)
0810 18:02:57.642989 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:57.642994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:57.643009 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:02:57.643837 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:57.643841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:57.643900 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:57.644010 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:57.645481 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:57.646038 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:57.647618 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:57.647852 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:57.647969 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:57.651328 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:57.651333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:57.651335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:57.651367 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:57.653229 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:57.653948 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:57.653952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:57.653954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:57.654090 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:57.654561 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:57.654566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:57.654568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:57.654857 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:57.654940 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:57.655519 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 18:02:57.655668 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:02:57.656800 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:57.641672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33573 (local address 127.28.83.66:49399)
0810 18:02:57.642307 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:57.642365 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:57.643604 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:57.644205 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:57.644212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:57.644621 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:57.645301 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:57.645318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:57.646185 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:57.646194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:57.647447 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:57.647458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:57.648013 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:57.648017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:57.648221 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:57.649171 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:57.649194 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:57.651117 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:57.653383 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:57.653389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:57.653434 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:57.653819 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:57.654228 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:57.654233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:57.654236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:57.654438 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:57.655001 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:57.655006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:57.655267 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 18:02:57.655461 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 18:02:57.655797 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":225,"threads_started":1}
I20260810 18:02:57.658146 8440 heartbeater.cc:347] Connected to a master server at 127.28.83.126:33573
I20260810 18:02:57.658468 8440 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:57.659401 8440 heartbeater.cc:511] Master 127.28.83.126:33573 requested a full tablet report, sending...
I20260810 18:02:57.660900 8088 ts_manager.cc:194] Registered new tserver with Master: f949591780ea41f385dac86fb239c2ff (127.28.83.66:37125)
I20260810 18:02:57.661614 8088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:49399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:02:57.681444 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:57469/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33573
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:02:57.885007 8445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:02:57.885574 8445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:02:57.885710 8445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:02:57.891969 8445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:02:57.892132 8445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:02:57.892272 8445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:02:57.900238 8445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.83.84:57469/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:33573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:02:57.902134 8445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:02:57.903884 8445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:02:57.919768 8453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:02:57.920252 8451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:57.921499 8445 server_base.cc:1034] running on GCE node
W20260810 18:02:57.921821 8450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:02:57.922384 8445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:02:57.923166 8445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:02:57.930631 8445 hybrid_clock.cc:648] HybridClock initialized: now 1786384977930108 us; error 528 us; skew 500 ppm
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:02:57.936395 8445 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:02:57.937909 8445 webserver.cc:460] Webserver started at http://127.28.83.67:46067/ using document root <none> and password file <none>
I20260810 18:02:57.938735 8445 fs_manager.cc:359] Metadata directory not provided
I20260810 18:02:57.938869 8445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:02:57.939167 8445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:02:57 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:02:57.982167 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b"
format_stamp: "Formatted at 2026-08-10 18:02:57 on dist-test-slave-srgv"
server_key: "fe8c4a44df734dceba78ba22fbacadb7"
server_key_iv: "0e7425db85406c4a676da8d7092ebe33"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:57.983310 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b"
format_stamp: "Formatted at 2026-08-10 18:02:57 on dist-test-slave-srgv"
server_key: "fe8c4a44df734dceba78ba22fbacadb7"
server_key_iv: "0e7425db85406c4a676da8d7092ebe33"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:57.989890 8445 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 18:02:58.021833 8460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:58.023824 8445 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260810 18:02:58.023985 8445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b"
format_stamp: "Formatted at 2026-08-10 18:02:57 on dist-test-slave-srgv"
server_key: "fe8c4a44df734dceba78ba22fbacadb7"
server_key_iv: "0e7425db85406c4a676da8d7092ebe33"
server_key_version: "kuduclusterkey@0"
I20260810 18:02:58.024245 8445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:02:58.051410 8445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:02:58.055590 8445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:02:58.055811 8445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:02:58.056674 8445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:02:58.057958 8445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:02:58.058027 8445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:58.058092 8445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:02:58.058122 8445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:02:58.085551 8445 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:35589
I20260810 18:02:58.086256 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:35589 every 8 connection(s)
I20260810 18:02:58.086889 8445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:02:58.093552 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 8445
I20260810 18:02:58.093736 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:02:58.094049 29005 external_mini_cluster.cc:1428] Setting key d4a6606ef55967e490529008d186879d
Aug 10 18:02:58 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:02:58.109016 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.092120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:59807 (local address 127.28.83.126:33573)
0810 18:02:58.092280 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:58.092285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:58.092305 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:02:58.093642 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:58.093647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:58.093708 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:58.093766 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:58.096774 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.097775 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.099744 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.100123 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.100225 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.104324 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:58.104328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:58.104331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:58.104365 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:58.106298 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:58.107012 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:58.107016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:58.107018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:58.107109 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:58.107564 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:58.107567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:58.107569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:58.107819 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:58.107899 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:58.108654 (+ 755us) server_negotiation.cc:299] Negotiation successful
0810 18:02:58.108820 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:02:58.111083 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.090759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33573 (local address 127.28.83.67:59807)
0810 18:02:58.091633 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:58.091732 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:58.093291 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:58.094822 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:58.094848 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:58.095367 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:02:58.096522 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.096546 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.097952 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.097956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.099555 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.099567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.100488 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.100492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.100770 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.101769 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:02:58.101791 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:02:58.104071 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:02:58.106468 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:58.106474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:58.106493 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:58.106866 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:58.107245 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:02:58.107248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:02:58.107250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:02:58.107448 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:02:58.107991 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:02:58.107999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:02:58.108348 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 18:02:58.109547 (+ 1199us) client_negotiation.cc:240] Negotiation successful
0810 18:02:58.109907 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":158,"threads_started":1}
I20260810 18:02:58.112782 8574 heartbeater.cc:347] Connected to a master server at 127.28.83.126:33573
I20260810 18:02:58.113027 8574 heartbeater.cc:464] Registering TS with master...
I20260810 18:02:58.113595 8574 heartbeater.cc:511] Master 127.28.83.126:33573 requested a full tablet report, sending...
I20260810 18:02:58.114846 8088 ts_manager.cc:194] Registered new tserver with Master: c8fce7ca7b8b4ceb9d06bc177fc0a39b (127.28.83.67:35589)
I20260810 18:02:58.115594 8088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:59807
I20260810 18:02:58.126199 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:02:58.140172 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.128724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.28.83.126:33573)
0810 18:02:58.128845 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:58.128850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:58.129538 (+ 688us) server_negotiation.cc:407] Connection header received
0810 18:02:58.129717 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:58.129720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:58.129766 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:58.129818 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:02:58.131218 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.132004 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.133913 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.134251 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.134321 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.136354 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:02:58.136358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:02:58.136361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:02:58.136389 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:02:58.138136 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:58.138833 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:58.138837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:58.138839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:58.138922 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:02:58.139232 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:02:58.139236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:02:58.139237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:02:58.139445 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:02:58.139542 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:58.139878 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 18:02:58.139997 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:02:58.144115 8088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:02:58.148878 8088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:02:58.190233 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.166858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42045 (local address 127.0.0.1:38960)
0810 18:02:58.171205 (+ 4347us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:58.171219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:58.171340 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:58.174143 (+ 2803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:58.174146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:58.174162 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:58.174305 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.174312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.182929 (+ 8617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.182934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.184376 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.184388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.189513 (+ 5125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.189517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.189735 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.189790 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 18:02:58.189909 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 18:02:58.190052 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4275,"thread_start_us":106,"threads_started":1}
I20260810 18:02:58.192173 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.162111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:37125 (local address 127.0.0.1:46604)
0810 18:02:58.167985 (+ 5874us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:58.168068 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:58.168207 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:58.173804 (+ 5597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:58.173806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:58.173822 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:58.173897 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.173902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.184974 (+ 11072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.184977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.186186 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.186198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.191739 (+ 5541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.191743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.191839 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.191862 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:02:58.191924 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:02:58.192002 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5668,"thread_start_us":164,"threads_started":1}
I20260810 18:02:58.195088 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.164792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:35589 (local address 127.0.0.1:58588)
0810 18:02:58.169577 (+ 4785us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:58.169595 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:58.169735 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:58.173478 (+ 3743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:58.173485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:58.173509 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:58.173683 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.173690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.178611 (+ 4921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.178614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.180156 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.180169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.193544 (+ 13375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.193548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.193646 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.193672 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:02:58.194841 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0810 18:02:58.194930 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4716,"thread_start_us":115,"threads_started":1}
I20260810 18:02:58.197310 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.167271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.28.83.65:42045)
0810 18:02:58.170009 (+ 2738us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:58.170018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:58.171436 (+ 1418us) server_negotiation.cc:407] Connection header received
0810 18:02:58.171513 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:58.171518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:58.171708 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:58.171842 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:58.174442 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.182678 (+ 8236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.186351 (+ 3673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.189172 (+ 2821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.189292 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.196688 (+ 7396us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:58.196799 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 18:02:58.197014 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2612,"thread_start_us":103,"threads_started":1}
I20260810 18:02:58.197921 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.163181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.28.83.66:37125)
0810 18:02:58.172805 (+ 9624us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:58.172812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:58.172836 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:02:58.172906 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:58.172911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:58.173096 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:58.173237 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:58.175711 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.184838 (+ 9127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.188121 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.191391 (+ 3270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.191511 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.191667 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:58.197611 (+ 5944us) server_negotiation.cc:299] Negotiation successful
0810 18:02:58.197742 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9510,"thread_start_us":175,"threads_started":1}
I20260810 18:02:58.198513 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.165293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.28.83.67:35589)
0810 18:02:58.170109 (+ 4816us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:58.170162 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:58.170196 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:02:58.170291 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:58.170298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:58.170585 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:58.170806 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:58.176973 (+ 6167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.178446 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.188433 (+ 9987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.193137 (+ 4704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.193269 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.193456 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:58.198185 (+ 4729us) server_negotiation.cc:299] Negotiation successful
0810 18:02:58.198307 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3227,"thread_start_us":178,"threads_started":1}
I20260810 18:02:58.202128 8372 tablet_service.cc:1467] Processing CreateTablet for tablet 3df79bcac8fd49ba93b5c73cb041c810 (DEFAULT_TABLE table=test-table [id=5de52bcaca254b90bd8d2406f46d6bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:02:58.204687 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df79bcac8fd49ba93b5c73cb041c810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:58.218004 8593 tablet_bootstrap.cc:492] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff: Bootstrap starting.
I20260810 18:02:58.217746 8226 tablet_service.cc:1467] Processing CreateTablet for tablet 3df79bcac8fd49ba93b5c73cb041c810 (DEFAULT_TABLE table=test-table [id=5de52bcaca254b90bd8d2406f46d6bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:02:58.218226 8508 tablet_service.cc:1467] Processing CreateTablet for tablet 3df79bcac8fd49ba93b5c73cb041c810 (DEFAULT_TABLE table=test-table [id=5de52bcaca254b90bd8d2406f46d6bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:02:58.219039 8226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df79bcac8fd49ba93b5c73cb041c810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:58.219343 8508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df79bcac8fd49ba93b5c73cb041c810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:02:58.221459 8593 tablet_bootstrap.cc:654] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:58.223249 8593 log.cc:824] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:58.225963 8593 tablet_bootstrap.cc:492] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff: No bootstrap required, opened a new log
I20260810 18:02:58.226194 8593 ts_tablet_manager.cc:1397] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260810 18:02:58.235601 8593 raft_consensus.cc:348] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } }
I20260810 18:02:58.236176 8593 raft_consensus.cc:374] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:58.236303 8593 raft_consensus.cc:729] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f949591780ea41f385dac86fb239c2ff, State: Initialized, Role: FOLLOWER
I20260810 18:02:58.236896 8593 consensus_queue.cc:260] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } }
I20260810 18:02:58.238314 8593 ts_tablet_manager.cc:1428] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff: Time spent starting tablet: real 0.012s user 0.006s sys 0.003s
I20260810 18:02:58.238729 8440 heartbeater.cc:503] Master 127.28.83.126:33573 was elected leader, sending a full tablet report...
I20260810 18:02:58.242842 8595 tablet_bootstrap.cc:492] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680: Bootstrap starting.
I20260810 18:02:58.244818 8596 tablet_bootstrap.cc:492] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b: Bootstrap starting.
I20260810 18:02:58.250005 8596 tablet_bootstrap.cc:654] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:58.251582 8596 log.cc:824] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:58.256752 8595 tablet_bootstrap.cc:654] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680: Neither blocks nor log segments found. Creating new log.
I20260810 18:02:58.259017 8596 tablet_bootstrap.cc:492] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b: No bootstrap required, opened a new log
I20260810 18:02:58.259313 8596 ts_tablet_manager.cc:1397] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260810 18:02:58.266404 8595 log.cc:824] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680: Log is configured to *not* fsync() on all Append() calls
I20260810 18:02:58.268272 8596 raft_consensus.cc:348] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } }
I20260810 18:02:58.268661 8596 raft_consensus.cc:374] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:58.268755 8596 raft_consensus.cc:729] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8fce7ca7b8b4ceb9d06bc177fc0a39b, State: Initialized, Role: FOLLOWER
I20260810 18:02:58.269380 8596 consensus_queue.cc:260] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } }
I20260810 18:02:58.270848 8596 ts_tablet_manager.cc:1428] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b: Time spent starting tablet: real 0.011s user 0.000s sys 0.008s
I20260810 18:02:58.271282 8574 heartbeater.cc:503] Master 127.28.83.126:33573 was elected leader, sending a full tablet report...
I20260810 18:02:58.275416 8303 heartbeater.cc:503] Master 127.28.83.126:33573 was elected leader, sending a full tablet report...
I20260810 18:02:58.276751 8595 tablet_bootstrap.cc:492] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680: No bootstrap required, opened a new log
I20260810 18:02:58.277038 8595 ts_tablet_manager.cc:1397] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.000s
I20260810 18:02:58.285171 8595 raft_consensus.cc:348] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } }
I20260810 18:02:58.285804 8595 raft_consensus.cc:374] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:02:58.285923 8595 raft_consensus.cc:729] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400899edd8b244139df5ebb50fadd680, State: Initialized, Role: FOLLOWER
I20260810 18:02:58.286559 8595 consensus_queue.cc:260] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } }
I20260810 18:02:58.288070 8595 ts_tablet_manager.cc:1428] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
W20260810 18:02:58.340600 8575 tablet.cc:2369] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 18:02:58 dist-test-slave-srgv krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384977, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
W20260810 18:02:58.388701 8441 tablet.cc:2369] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:02:58.447144 8600 raft_consensus.cc:482] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:02:58.447420 8600 raft_consensus.cc:504] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } }
I20260810 18:02:58.448941 8600 leader_election.cc:288] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f949591780ea41f385dac86fb239c2ff (127.28.83.66:37125), 400899edd8b244139df5ebb50fadd680 (127.28.83.65:42045)
I20260810 18:02:58.455595 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.449457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:37125 (local address 127.28.83.67:48345)
0810 18:02:58.450465 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:58.450482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:58.450582 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:58.451570 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:58.451573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:58.451591 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:58.451750 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.451756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.452800 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.452803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.454054 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.454065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.455135 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.455139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.455266 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.455288 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:02:58.455360 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:02:58.455438 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921}
I20260810 18:02:58.456506 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.450954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:48345 (local address 127.28.83.66:37125)
0810 18:02:58.451120 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:58.451126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:58.451144 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:02:58.451200 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:58.451203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:58.451251 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:58.451338 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:58.451886 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.452654 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.454225 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.454887 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.455004 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.455074 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:58.456310 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0810 18:02:58.456391 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:02:58.457543 8393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df79bcac8fd49ba93b5c73cb041c810" candidate_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f949591780ea41f385dac86fb239c2ff" is_pre_election: true
I20260810 18:02:58.457923 8393 raft_consensus.cc:2454] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8fce7ca7b8b4ceb9d06bc177fc0a39b in term 0.
I20260810 18:02:58.458806 8461 leader_election.cc:302] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8fce7ca7b8b4ceb9d06bc177fc0a39b, f949591780ea41f385dac86fb239c2ff; no voters:
I20260810 18:02:58.459169 8600 raft_consensus.cc:2781] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:02:58.459261 8600 raft_consensus.cc:482] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:02:58.459306 8600 raft_consensus.cc:3037] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:58.461014 8600 raft_consensus.cc:504] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } }
I20260810 18:02:58.461757 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.449876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42045 (local address 127.28.83.67:36683)
0810 18:02:58.455787 (+ 5911us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:58.455803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:58.455883 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:58.456675 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:02:58.456679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:02:58.456697 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:02:58.456769 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.456774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.457913 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.457916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.460258 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:02:58.460269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.461265 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.461269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:02:58.461359 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.461383 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:02:58.461476 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 18:02:58.461557 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5836,"thread_start_us":185,"threads_started":1}
I20260810 18:02:58.462220 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.451127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:36683 (local address 127.28.83.65:42045)
0810 18:02:58.451510 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:58.451515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:58.455968 (+ 4453us) server_negotiation.cc:407] Connection header received
0810 18:02:58.456026 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:58.456029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:58.456134 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:58.456201 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:02:58.456902 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.457765 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.460403 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.461022 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.461154 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.461205 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:58.461993 (+ 788us) server_negotiation.cc:299] Negotiation successful
0810 18:02:58.462095 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260810 18:02:58.462199 8393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df79bcac8fd49ba93b5c73cb041c810" candidate_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f949591780ea41f385dac86fb239c2ff"
I20260810 18:02:58.462338 8393 raft_consensus.cc:3037] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:58.463990 8393 raft_consensus.cc:2454] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8fce7ca7b8b4ceb9d06bc177fc0a39b in term 1.
I20260810 18:02:58.464668 8256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df79bcac8fd49ba93b5c73cb041c810" candidate_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400899edd8b244139df5ebb50fadd680" is_pre_election: true
I20260810 18:02:58.465001 8256 raft_consensus.cc:2454] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8fce7ca7b8b4ceb9d06bc177fc0a39b in term 0.
I20260810 18:02:58.466033 8461 leader_election.cc:302] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8fce7ca7b8b4ceb9d06bc177fc0a39b, f949591780ea41f385dac86fb239c2ff; no voters:
I20260810 18:02:58.466480 8600 leader_election.cc:288] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [CANDIDATE]: Term 1 election: Requested vote from peers f949591780ea41f385dac86fb239c2ff (127.28.83.66:37125), 400899edd8b244139df5ebb50fadd680 (127.28.83.65:42045)
I20260810 18:02:58.466590 8600 raft_consensus.cc:2781] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:02:58.466663 8600 raft_consensus.cc:686] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 1 LEADER]: Becoming Leader. State: Replica: c8fce7ca7b8b4ceb9d06bc177fc0a39b, State: Running, Role: LEADER
I20260810 18:02:58.467032 8600 consensus_queue.cc:237] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } }
I20260810 18:02:58.468544 8256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df79bcac8fd49ba93b5c73cb041c810" candidate_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400899edd8b244139df5ebb50fadd680"
I20260810 18:02:58.468683 8256 raft_consensus.cc:3037] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:02:58.470645 8256 raft_consensus.cc:2454] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8fce7ca7b8b4ceb9d06bc177fc0a39b in term 1.
I20260810 18:02:58.488019 8088 catalog_manager.cc:5495] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b reported cstate change: term changed from 0 to 1, leader changed from <none> to c8fce7ca7b8b4ceb9d06bc177fc0a39b (127.28.83.67). New cstate: current_term: 1 leader_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8fce7ca7b8b4ceb9d06bc177fc0a39b" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 } health_report { overall_health: UNKNOWN } } }
W20260810 18:02:58.494724 8304 tablet.cc:2369] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:02:58.542874 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.537039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.28.83.67:35589)
0810 18:02:58.537232 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 18:02:58.537237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:02:58.537283 (+ 46us) server_negotiation.cc:407] Connection header received
0810 18:02:58.537513 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:02:58.537516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:02:58.537584 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:02:58.537694 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:02:58.538235 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.539007 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.540360 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:02:58.540693 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:02:58.540799 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:02:58.541269 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:02:58.541886 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:02:58.542012 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:02:58.542513 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 18:02:58.542581 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 18:02:58.552103 8256 raft_consensus.cc:1261] T 3df79bcac8fd49ba93b5c73cb041c810 P 400899edd8b244139df5ebb50fadd680 [term 1 FOLLOWER]: Refusing update from remote peer c8fce7ca7b8b4ceb9d06bc177fc0a39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:02:58.553222 8605 consensus_queue.cc:1035] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "400899edd8b244139df5ebb50fadd680" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:02:58.561510 8393 raft_consensus.cc:1261] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [term 1 FOLLOWER]: Refusing update from remote peer c8fce7ca7b8b4ceb9d06bc177fc0a39b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:02:58.562367 8605 consensus_queue.cc:1035] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f949591780ea41f385dac86fb239c2ff" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 37125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:02:58.572149 8608 mvcc.cc:205] Tried to move back new op lower bound from 7317032872138887168 to 7317032871818551296. Current Snapshot: MvccSnapshot[applied={T|T < 7317032872138887168}]
I20260810 18:02:58.572487 8609 mvcc.cc:205] Tried to move back new op lower bound from 7317032872138887168 to 7317032871818551296. Current Snapshot: MvccSnapshot[applied={T|T < 7317032872138887168 or (T in {7317032872138887168})}]
I20260810 18:02:58.581589 8086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:02:58.581790 8086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:02:58.585028 8086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01dc61bd58264c3289fc6cdfd6cf0a57: Sending DeleteTablet for 3 replicas of tablet 3df79bcac8fd49ba93b5c73cb041c810
I20260810 18:02:58.587014 8372 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df79bcac8fd49ba93b5c73cb041c810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:02:58 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:46604
I20260810 18:02:58.587647 8617 tablet_replica.cc:330] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff: stopping tablet replica
I20260810 18:02:58.588279 8617 raft_consensus.cc:2229] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:02:58.588253 8508 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df79bcac8fd49ba93b5c73cb041c810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:02:58 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:58588
I20260810 18:02:58.588742 8617 raft_consensus.cc:2258] T 3df79bcac8fd49ba93b5c73cb041c810 P f949591780ea41f385dac86fb239c2ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:02:58.590631 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 8169
W20260810 18:02:58.599542 8073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:42045 (error 108)
I20260810 18:02:58.600311 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 8309
I20260810 18:02:58.601289 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:02:58.600458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42045 (local address 127.0.0.1:38962)
0810 18:02:58.601103 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:58.601182 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:42045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536}
W20260810 18:02:58.602298 8073 catalog_manager.cc:4584] TS 400899edd8b244139df5ebb50fadd680 (127.28.83.65:42045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3df79bcac8fd49ba93b5c73cb041c810: Network error: Client connection negotiation failed: client connection to 127.28.83.65:42045: connect: Connection refused (error 111)
I20260810 18:02:58.605715 8618 tablet_replica.cc:330] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b: stopping tablet replica
I20260810 18:02:58.612116 8618 raft_consensus.cc:2229] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:02:58.612685 8618 raft_consensus.cc:2258] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 18:02:58.614467 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:58.613834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:37125 (local address 127.0.0.1:46614)
0810 18:02:58.614010 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:58.614025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:58.614113 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:58.614346 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.66:37125: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 18:02:58.614903 8618 ts_tablet_manager.cc:1905] T 3df79bcac8fd49ba93b5c73cb041c810 P c8fce7ca7b8b4ceb9d06bc177fc0a39b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:02:58.615406 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 8445
W20260810 18:02:58.625885 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:02:58.625256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:35589 (local address 127.0.0.1:58610)
0810 18:02:58.625467 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 18:02:58.625483 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:02:58.625592 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:02:58.625767 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.67:35589: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 18:02:58.626752 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 8055
I20260810 18:02:58.633863 29005 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 18:02:58.978720 29005 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 18:02:58.978901 29005 mini_ranger.cc:66] Stopping Ranger...
I20260810 18:02:59.324133 29005 mini_ranger.cc:68] Stopped Ranger
2026-08-10 18:02:59.324 UTC [7637] LOG: received smart shutdown request
2026-08-10 18:02:59.327 UTC [7637] LOG: background worker "logical replication launcher" (PID 7647) exited with exit code 1
2026-08-10 18:02:59.328 UTC [7642] LOG: shutting down
2026-08-10 18:02:59.499 UTC [7637] LOG: database system is shut down
2026-08-10T18:02:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52370 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:02:59 dist-test-slave-srgv krb5kdc[8629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:02:59 dist-test-slave-srgv krb5kdc[8629](info): set up 2 sockets
Aug 10 18:02:59 dist-test-slave-srgv krb5kdc[8629](info): commencing operation
krb5kdc: starting...
W20260810 18:03:00.706806 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:03:00 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786384980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:03:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/rangeradmin_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/rangeradmin_127.28.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/rangerlookup_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/rangerlookup_127.28.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/HTTP_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/HTTP_127.28.83.84@KRBTEST.COM.keytab.
I20260810 18:03:00.880694 29005 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5EWyHF/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/postgres -l logfile start
2026-08-10 18:03:03.381 UTC [8659] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 18:03:03.381 UTC [8659] LOG: listening on IPv4 address "127.28.83.84", port 51091
2026-08-10 18:03:03.385 UTC [8659] LOG: listening on Unix socket "/tmp/.s.PGSQL.51091"
2026-08-10 18:03:03.408 UTC [8662] LOG: database system was shut down at 2026-08-10 18:03:01 UTC
2026-08-10 18:03:03.411 UTC [8659] LOG: database system is ready to accept connections
I20260810 18:03:04.383913 29005 mini_postgres.cc:96] Postgres bound to 51091
2026-08-10 18:03:04.391 UTC [8670] FATAL: database "slave" does not exist
127.28.83.84:51091 - accepting connections
I20260810 18:03:04.391907 29005 mini_ranger.cc:161] Starting Ranger...
I20260810 18:03:04.407778 29005 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 18:03:04.878676 29005 mini_ranger.cc:87] Created ranger Postgres database
I20260810 18:03:04.878770 29005 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-admin
2026-08-10 18:03:05,161 [I] DB FLAVOR :POSTGRES
2026-08-10 18:03:05,162 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:03:05,162 [I] Checking connection..
2026-08-10 18:03:05,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 18:03:05,420 [I] Checking connection passed.
2026-08-10 18:03:05,420 [I] --------- Verifying version history table ---------
2026-08-10 18:03:05,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:03:05,660 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:03:05,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:03:05,898 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:03:05,898 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 18:03:05,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 18:03:06.140 UTC [8734] WARNING: there is no transaction in progress
2026-08-10 18:03:06,147 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 18:03:06,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:03:06,410 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 18:03:06,411 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:03:06,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 18:03:06,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 18:03:06,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-srgv.c.gcp-upstream.internal','N') ;"
2026-08-10 18:03:07,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:03:07,361 [I] Table x_portal_user does not exist in database ranger
2026-08-10 18:03:07,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:03:07,605 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 18:03:07,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:03:07,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:03:08,099 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 18:03:08,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 18:03:09.016 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.032 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.048 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.063 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.079 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.096 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.149 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.155 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.170 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.180 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.189 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.204 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.213 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.219 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.226 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09.232 UTC [8842] WARNING: there is no transaction in progress
2026-08-10 18:03:09,832 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 18:03:09,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 18:03:10,085 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 18:03:10,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:03:10,355 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 18:03:10,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:03:10,618 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 18:03:10,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:03:10,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:03:11,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-srgv.c.gcp-upstream.internal';"
2026-08-10 18:03:11,363 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 18:03:11,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 18:03:11,607 [I] DB_PATCHES have already been applied
I20260810 18:03:11.613073 29005 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-admin:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5EWyHF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 18:03:11.613135 29005 mini_ranger.cc:193] Using host: 127.28.83.84
I20260810 18:03:11.616848 29005 mini_ranger.cc:227] Ranger admin URL: http://127.28.83.84:54233
Aug 10, 2026 6:03:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:03:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:03:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:03:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:03:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:03:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:03:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 18:03:12.749013 29005 mini_ranger.cc:160] Time spent starting Ranger: real 8.357s user 0.000s sys 0.011s
Aug 10, 2026 6:03:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:03:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 6:03:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:03:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:03:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 18:03:34.480363 29005 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/rangerkms_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/rangerkms_127.28.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.83.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.83.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.83.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/HTTP_127.28.83.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.83.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/HTTP_127.28.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 18:03:34 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385014, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 18:03:34.570432 29005 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 18:03:34.580104 29005 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 18:03:35.402181 29005 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 18:03:35.402281 29005 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-kms
I20260810 18:03:35.402304 29005 mini_ranger_kms.cc:227] Using postgres at 127.28.83.84:51091
2026-08-10 18:03:35,578 [I] DB FLAVOR :POSTGRES
2026-08-10 18:03:35,578 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:03:35,578 [I] Checking connection
2026-08-10 18:03:35,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 18:03:35,829 [I] connection success
2026-08-10 18:03:35,830 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 18:03:35,830 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 18:03:35,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 18:03:36,098 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 18:03:36,098 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:03:36,098 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 18:03:36,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.83.84:51091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 18:03:36,381 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 18:03:36.882241 29005 mini_ranger_kms.cc:310] Created kms service
I20260810 18:03:37.078064 29005 mini_ranger_kms.cc:326] Created kudu user
I20260810 18:03:37.123903 29005 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 18:03:37 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385014, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:37.509243 29005 mini_ranger_kms.cc:384] Added ranger policy
I20260810 18:03:37.509395 29005 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-kms:/tmp/dist-test-task5EWyHF/build/debug/bin/postgresql.jar:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5EWyHF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 18:03:37.509454 29005 mini_ranger_kms.cc:240] Using host: 127.28.83.84
I20260810 18:03:37.513572 29005 mini_ranger_kms.cc:276] Ranger KMS PID: 9022
I20260810 18:03:37.513674 29005 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.83.84:60607
Aug 10, 2026 6:03:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:03:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:03:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:03:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:03:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:03:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 18:03:39.815266 29005 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.245s user 0.010s sys 0.150s
I20260810 18:03:39.815389 29005 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 18:03:39.815454 29005 mini_ranger_kms.cc:401] 127.28.83.84:60607/kms/v1/keys
Aug 10 18:03:39 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385019, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.83.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 18:03:40 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385019, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.83.84@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
Aug 10 18:03:40 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385019, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.83.84@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
Aug 10 18:03:43 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:03:43.041206 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40035
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:40035
--ranger_config_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:60607/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:43.149899 9073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:43.150172 9073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:43.150255 9073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:43.154225 9073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:03:43.154295 9073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:43.154330 9073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:43.154382 9073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:03:43.154421 9073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:03:43.159772 9073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.83.84:60607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:40035
--ranger_config_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:43.160768 9073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:43.161815 9073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:43.171192 9081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:43.171200 9079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:43.171209 9078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:43.171355 9073 server_base.cc:1034] running on GCE node
I20260810 18:03:43.172043 9073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:43.173061 9073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:43.174247 9073 hybrid_clock.cc:648] HybridClock initialized: now 1786385023174208 us; error 57 us; skew 500 ppm
Aug 10 18:03:43 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:43.182301 9073 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:03:43.183534 9073 webserver.cc:460] Webserver started at http://127.28.83.126:35751/ using document root <none> and password file <none>
I20260810 18:03:43.184262 9073 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:43.184358 9073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:43.184614 9073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:03:43 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:43.238848 9073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "511d983d3dc5425da8dbd03ad943d05d"
format_stamp: "Formatted at 2026-08-10 18:03:43 on dist-test-slave-srgv"
server_key: "41124c939937eea5c01a8b11946e143d"
server_key_iv: "42b2cb124894db689362e192ee4b0f59"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:43.239481 9073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "511d983d3dc5425da8dbd03ad943d05d"
format_stamp: "Formatted at 2026-08-10 18:03:43 on dist-test-slave-srgv"
server_key: "41124c939937eea5c01a8b11946e143d"
server_key_iv: "42b2cb124894db689362e192ee4b0f59"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:43.243492 9073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:03:43.260828 9088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:43.262229 9073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:03:43.262359 9073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "511d983d3dc5425da8dbd03ad943d05d"
format_stamp: "Formatted at 2026-08-10 18:03:43 on dist-test-slave-srgv"
server_key: "41124c939937eea5c01a8b11946e143d"
server_key_iv: "42b2cb124894db689362e192ee4b0f59"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:43.262492 9073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:43.290331 9073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:43.293242 9073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:43.293511 9073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:43.301441 9073 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:40035
I20260810 18:03:43.301462 9141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:40035 every 8 connection(s)
I20260810 18:03:43.302412 9073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:03:43.305122 9142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:43.309471 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 9073
I20260810 18:03:43.309595 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:03:43.309872 29005 external_mini_cluster.cc:1428] Setting key 6b3866b9b31dc48fea30a13bbe443e17
I20260810 18:03:43.310731 9142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Bootstrap starting.
I20260810 18:03:43.313573 9142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:43.314448 9142 log.cc:824] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:43.316428 9142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: No bootstrap required, opened a new log
Aug 10 18:03:43 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:43.320551 9142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } }
I20260810 18:03:43.320837 9142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:43.320936 9142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 511d983d3dc5425da8dbd03ad943d05d, State: Initialized, Role: FOLLOWER
I20260810 18:03:43.321449 9142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } }
I20260810 18:03:43.321609 9142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:03:43.321691 9142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:03:43.321816 9142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:43.322691 9142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } }
I20260810 18:03:43.323088 9142 leader_election.cc:302] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 511d983d3dc5425da8dbd03ad943d05d; no voters:
I20260810 18:03:43.323490 9142 leader_election.cc:288] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:03:43.323788 9147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:03:43.324059 9147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 1 LEADER]: Becoming Leader. State: Replica: 511d983d3dc5425da8dbd03ad943d05d, State: Running, Role: LEADER
I20260810 18:03:43.324868 9142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:03:43.324406 9147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } }
I20260810 18:03:43.325987 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:43.311312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.28.83.126:40035)
0810 18:03:43.311807 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:43.311827 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:43.311870 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:03:43.312708 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:43.312732 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:43.313121 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:43.313511 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:43.314431 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:43.315689 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:43.316840 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:43.317535 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:43.317721 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:43.320739 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:43.320745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:43.320761 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:43.320792 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:43.322829 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:43.323316 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:43.323322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:43.323328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:43.323470 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:43.323769 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:43.323772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:43.323774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:43.324367 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:43.324550 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:43.324810 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 18:03:43.325070 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":195,"threads_started":1}
I20260810 18:03:43.328126 9148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 511d983d3dc5425da8dbd03ad943d05d. Latest consensus state: current_term: 1 leader_uuid: "511d983d3dc5425da8dbd03ad943d05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } } }
I20260810 18:03:43.328250 9148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:43.328760 9162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:03:43.328954 9149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "511d983d3dc5425da8dbd03ad943d05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } } }
I20260810 18:03:43.329052 9149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:43.329248 9142 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 18:03:43.331729 9162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:03:43.342873 9162 catalog_manager.cc:1347] Generated new cluster ID: 38ed272f42cf4d30a94ccf21ada01303
I20260810 18:03:43.342954 9162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:03:43.364864 9162 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:03:43.366858 9162 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:03:43.380576 9162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Generated new TSK 0
I20260810 18:03:43.381551 9162 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:03:44.137959 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:60607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40035
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:44.258536 9188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:44.258795 9188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:44.258839 9188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:44.263513 9188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:44.263577 9188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:44.263707 9188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:03:44.269574 9188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.83.84:60607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:44.270872 9188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:44.272147 9188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:44.284528 9196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:44.285099 9197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:44.285578 9199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:44.294443 9188 server_base.cc:1034] running on GCE node
I20260810 18:03:44.294930 9188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:44.295678 9188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:44.296867 9188 hybrid_clock.cc:648] HybridClock initialized: now 1786385024296842 us; error 44 us; skew 500 ppm
Aug 10 18:03:44 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:44.299801 9188 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:03:44.300864 9188 webserver.cc:460] Webserver started at http://127.28.83.65:42143/ using document root <none> and password file <none>
I20260810 18:03:44.301600 9188 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:44.301716 9188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:44.301947 9188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:03:44 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:44.326917 9188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "01d4cde497364fe48725c68c243fc250"
format_stamp: "Formatted at 2026-08-10 18:03:44 on dist-test-slave-srgv"
server_key: "574ba20878fa2604c92f3963ea208e8a"
server_key_iv: "d5d08ae9d17ac133f4af8be9bec866e0"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:44.328006 9188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "01d4cde497364fe48725c68c243fc250"
format_stamp: "Formatted at 2026-08-10 18:03:44 on dist-test-slave-srgv"
server_key: "574ba20878fa2604c92f3963ea208e8a"
server_key_iv: "d5d08ae9d17ac133f4af8be9bec866e0"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:44.332224 9188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:03:44.359524 9206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:44.366629 9188 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.002s sys 0.000s
I20260810 18:03:44.366832 9188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "01d4cde497364fe48725c68c243fc250"
format_stamp: "Formatted at 2026-08-10 18:03:44 on dist-test-slave-srgv"
server_key: "574ba20878fa2604c92f3963ea208e8a"
server_key_iv: "d5d08ae9d17ac133f4af8be9bec866e0"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:44.366991 9188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:44.395855 9188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:44.399854 9188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:44.400112 9188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:44.400870 9188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:44.402169 9188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:44.402307 9188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:44.402443 9188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:44.402545 9188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:44.416868 9188 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:32971
I20260810 18:03:44.418232 9188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:03:44.419199 9322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:32971 every 8 connection(s)
I20260810 18:03:44.426457 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 9188
I20260810 18:03:44.426580 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:03:44.426883 29005 external_mini_cluster.cc:1428] Setting key 7d61882252d00c2ee3051349c00aa4a0
Aug 10 18:03:44 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:44.438560 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:44.421663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:55499 (local address 127.28.83.126:40035)
0810 18:03:44.423673 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:44.423678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:44.424154 (+ 476us) server_negotiation.cc:407] Connection header received
0810 18:03:44.425582 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:44.425587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:44.425647 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:44.425796 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:44.427609 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:44.428396 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:44.430192 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:44.430569 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:44.430646 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:44.433812 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:44.433816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:44.433819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:44.433848 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:44.435844 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:44.436548 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:44.436551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:44.436553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:44.436645 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:44.437130 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:44.437133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:44.437135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:44.437380 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:44.437487 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:44.438205 (+ 718us) server_negotiation.cc:299] Negotiation successful
0810 18:03:44.438355 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1918,"thread_start_us":92,"threads_started":1}
I20260810 18:03:44.440534 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:44.422187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40035 (local address 127.28.83.65:55499)
0810 18:03:44.423954 (+ 1767us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:44.424011 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:44.425293 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:44.426382 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:44.426405 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:44.426769 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:44.427448 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:44.427471 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:44.428553 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:44.428556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:44.430031 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:44.430045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:44.430862 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:44.430884 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:44.431083 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:44.431840 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:44.431861 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:44.433602 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:44.436004 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:44.436013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:44.436031 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:44.436404 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:44.436761 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:44.436763 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:44.436765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:44.436993 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:44.437557 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:44.437564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:44.437894 (+ 330us) client_negotiation.cc:769] Sending connection context
0810 18:03:44.439039 (+ 1145us) client_negotiation.cc:240] Negotiation successful
0810 18:03:44.439442 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543,"thread_start_us":159,"threads_started":1}
I20260810 18:03:44.442368 9323 heartbeater.cc:347] Connected to a master server at 127.28.83.126:40035
I20260810 18:03:44.442620 9323 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:44.443215 9323 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
I20260810 18:03:44.445092 9105 ts_manager.cc:194] Registered new tserver with Master: 01d4cde497364fe48725c68c243fc250 (127.28.83.65:32971)
I20260810 18:03:44.446652 9105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:55499
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:03:44.473292 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:60607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40035
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:03:44 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 18:03:44.659261 9329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:44.659729 9329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:44.659869 9329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:44.666134 9329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:44.666278 9329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:44.666410 9329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:03:44.674413 9329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.83.84:60607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:44.675980 9329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:44.677336 9329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:44.692700 9336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:44.693272 9335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:44.694870 9329 server_base.cc:1034] running on GCE node
W20260810 18:03:44.697611 9338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:44.698200 9329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:44.698963 9329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:44.703650 9329 hybrid_clock.cc:648] HybridClock initialized: now 1786385024703027 us; error 631 us; skew 500 ppm
Aug 10 18:03:44 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:44.708217 9329 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:03:44.709975 9329 webserver.cc:460] Webserver started at http://127.28.83.66:39631/ using document root <none> and password file <none>
I20260810 18:03:44.710724 9329 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:44.710862 9329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:44.711133 9329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:03:44 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:44.735110 9329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "ec93b5531fa74d33bf5370ce9240cdc5"
format_stamp: "Formatted at 2026-08-10 18:03:44 on dist-test-slave-srgv"
server_key: "65928ecca3f42633cb6cf7d6c1848dfb"
server_key_iv: "bc50b90647a24c4cd226253ac473856a"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:44.736119 9329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "ec93b5531fa74d33bf5370ce9240cdc5"
format_stamp: "Formatted at 2026-08-10 18:03:44 on dist-test-slave-srgv"
server_key: "65928ecca3f42633cb6cf7d6c1848dfb"
server_key_iv: "bc50b90647a24c4cd226253ac473856a"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:44.741978 9329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 18:03:44.763564 9345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:44.765329 9329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 18:03:44.765482 9329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "ec93b5531fa74d33bf5370ce9240cdc5"
format_stamp: "Formatted at 2026-08-10 18:03:44 on dist-test-slave-srgv"
server_key: "65928ecca3f42633cb6cf7d6c1848dfb"
server_key_iv: "bc50b90647a24c4cd226253ac473856a"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:44.765615 9329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:44.801383 9329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:44.805192 9329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:44.807087 9329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:44.808451 9329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:44.809926 9329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:44.810045 9329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:44.810140 9329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:44.810201 9329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:44.842878 9329 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:44763
I20260810 18:03:44.844228 9329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:03:44.846282 9459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:44763 every 8 connection(s)
I20260810 18:03:44.852253 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 9329
I20260810 18:03:44.852387 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:03:44.852674 29005 external_mini_cluster.cc:1428] Setting key 4fb8a4e689de0c19e146ddfcebaea7d1
Aug 10 18:03:44 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:44.879411 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:44.848251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:39585 (local address 127.28.83.126:40035)
0810 18:03:44.848421 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:44.848425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:44.849726 (+ 1301us) server_negotiation.cc:407] Connection header received
0810 18:03:44.851188 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:44.851191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:44.851256 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:44.851342 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:44.853087 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:44.854297 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:44.856712 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:44.857036 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:44.857128 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:44.864008 (+ 6880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:44.864012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:44.864015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:44.864068 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:44.873470 (+ 9402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:44.877316 (+ 3846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:44.877321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:44.877323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:44.877471 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:44.877973 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:44.877976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:44.877978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:44.878236 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:44.878314 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:44.879064 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 18:03:44.879235 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:03:44.883011 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:44.848758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40035 (local address 127.28.83.66:39585)
0810 18:03:44.849541 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:44.849603 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:44.850894 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:44.851538 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:44.851547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:44.851968 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:44.852853 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:44.852876 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:44.854561 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:44.854564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:44.856567 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:44.856578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:44.857332 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:44.857336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:44.857582 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:44.861790 (+ 4208us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:44.861814 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:44.863758 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:44.873721 (+ 9963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:44.873729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:44.873750 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:44.874261 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:44.877615 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:44.877618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:44.877620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:44.877848 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:44.878409 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:44.878417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:44.878775 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 18:03:44.879925 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0810 18:03:44.880339 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":272,"threads_started":1}
I20260810 18:03:44.884764 9460 heartbeater.cc:347] Connected to a master server at 127.28.83.126:40035
I20260810 18:03:44.885155 9460 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:44.885973 9460 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
I20260810 18:03:44.887195 9105 ts_manager.cc:194] Registered new tserver with Master: ec93b5531fa74d33bf5370ce9240cdc5 (127.28.83.66:44763)
I20260810 18:03:44.887933 9105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:39585
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:03:44.912133 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:60607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40035
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:45.084468 9465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:45.084925 9465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:45.085036 9465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:45.091164 9465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:45.091317 9465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:45.091454 9465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:03:45.099337 9465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.83.84:60607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:45.100907 9465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:45.102321 9465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:45.115223 9472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:45.115607 9474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:45.115624 9471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:45.116926 9465 server_base.cc:1034] running on GCE node
I20260810 18:03:45.117606 9465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:45.118376 9465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:45.119557 9465 hybrid_clock.cc:648] HybridClock initialized: now 1786385025119526 us; error 48 us; skew 500 ppm
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:45.123504 9465 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:03:45.124775 9465 webserver.cc:460] Webserver started at http://127.28.83.67:43165/ using document root <none> and password file <none>
I20260810 18:03:45.125552 9465 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:45.125741 9465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:45.126011 9465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:45.147883 9465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "1b508c31280c4113b3e1afea0998a02a"
format_stamp: "Formatted at 2026-08-10 18:03:45 on dist-test-slave-srgv"
server_key: "cbd994c030210bfc895baa2afa1b1983"
server_key_iv: "d69459c75b3dc072556fea8a6a6479f3"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:45.149871 9465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "1b508c31280c4113b3e1afea0998a02a"
format_stamp: "Formatted at 2026-08-10 18:03:45 on dist-test-slave-srgv"
server_key: "cbd994c030210bfc895baa2afa1b1983"
server_key_iv: "d69459c75b3dc072556fea8a6a6479f3"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:45.158946 9465 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.000s
I20260810 18:03:45.180997 9481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:45.182801 9465 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260810 18:03:45.182940 9465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "1b508c31280c4113b3e1afea0998a02a"
format_stamp: "Formatted at 2026-08-10 18:03:45 on dist-test-slave-srgv"
server_key: "cbd994c030210bfc895baa2afa1b1983"
server_key_iv: "d69459c75b3dc072556fea8a6a6479f3"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:45.183081 9465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:45.201301 9465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:45.205085 9465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:45.205375 9465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:45.206149 9465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:45.207340 9465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:45.207475 9465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:45.207578 9465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:45.207643 9465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:45.219772 9465 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:40051
I20260810 18:03:45.223146 9465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:03:45.228554 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 9465
I20260810 18:03:45.228680 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:03:45.228968 29005 external_mini_cluster.cc:1428] Setting key e1f3beea1a0b21d6a3718000d03133a9
I20260810 18:03:45.224980 9595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:40051 every 8 connection(s)
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:45.245324 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:45.229365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:59451 (local address 127.28.83.126:40035)
0810 18:03:45.229702 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:45.229707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:45.230854 (+ 1147us) server_negotiation.cc:407] Connection header received
0810 18:03:45.232207 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:45.232211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:45.232268 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:45.232317 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:45.233906 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:45.234699 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:45.236231 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:45.236519 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:45.236587 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:45.239667 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:45.239671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:45.239674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:45.239700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:45.241467 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:45.242125 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:45.242128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:45.242130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:45.242221 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:45.242626 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:45.242629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:45.242630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:45.242936 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:45.243040 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:45.245023 (+ 1983us) server_negotiation.cc:299] Negotiation successful
0810 18:03:45.245155 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260810 18:03:45.246539 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:45.227277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40035 (local address 127.28.83.67:59451)
0810 18:03:45.230707 (+ 3430us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:45.230764 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:45.231953 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:45.232715 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:45.232723 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:45.233071 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:45.233744 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:45.233764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:45.234852 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:45.234855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:45.236087 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:45.236098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:45.236788 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:45.236792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:45.236986 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:45.237720 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:45.237740 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:45.239470 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:45.241599 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:45.241605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:45.241620 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:45.241974 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:45.242340 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:45.242343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:45.242345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:45.242527 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:45.244524 (+ 1997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:45.244530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:45.244770 (+ 240us) client_negotiation.cc:769] Sending connection context
0810 18:03:45.244962 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 18:03:45.245288 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3215,"thread_start_us":160,"threads_started":1}
I20260810 18:03:45.247988 9596 heartbeater.cc:347] Connected to a master server at 127.28.83.126:40035
I20260810 18:03:45.248229 9596 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:45.248696 9596 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
I20260810 18:03:45.249724 9105 ts_manager.cc:194] Registered new tserver with Master: 1b508c31280c4113b3e1afea0998a02a (127.28.83.67:40051)
I20260810 18:03:45.250440 9105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:59451
I20260810 18:03:45.254874 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:03:45.255720 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 9188
I20260810 18:03:45.271773 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 9329
I20260810 18:03:45.279527 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 9465
I20260810 18:03:45.297748 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 9073
I20260810 18:03:45.306010 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40035
--webserver_interface=127.28.83.126
--webserver_port=35751
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:40035
--ranger_config_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:60607/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.83.84:60607/kms with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:45.415736 9599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:45.415999 9599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:45.416038 9599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:45.419943 9599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:03:45.420003 9599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:45.420022 9599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:45.420037 9599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:03:45.420053 9599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:03:45.425465 9599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.83.84:60607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:40035
--ranger_config_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=35751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:45.426457 9599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:45.427425 9599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:45.435725 9604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:45.435752 9607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:45.435961 9599 server_base.cc:1034] running on GCE node
W20260810 18:03:45.436137 9605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:45.436723 9599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:45.437870 9599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:45.439023 9599 hybrid_clock.cc:648] HybridClock initialized: now 1786385025439009 us; error 25 us; skew 500 ppm
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:45.441748 9599 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:03:45.442567 9599 webserver.cc:460] Webserver started at http://127.28.83.126:35751/ using document root <none> and password file <none>
I20260810 18:03:45.443042 9599 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:45.443110 9599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:45.520030 9599 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:03:45.522740 9615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:45.524185 9599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:03:45.524341 9599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "511d983d3dc5425da8dbd03ad943d05d"
format_stamp: "Formatted at 2026-08-10 18:03:43 on dist-test-slave-srgv"
server_key: "41124c939937eea5c01a8b11946e143d"
server_key_iv: "42b2cb124894db689362e192ee4b0f59"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:45.524933 9599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:45.540808 9599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:45.543889 9599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:45.544100 9599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:45.551189 9599 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:40035
I20260810 18:03:45.551226 9667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:40035 every 8 connection(s)
I20260810 18:03:45.552228 9599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:03:45.553905 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 9599
I20260810 18:03:45.554347 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:60607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:32971
--local_ip_for_outbound_sockets=127.28.83.65
--tserver_master_addrs=127.28.83.126:40035
--webserver_port=42143
--webserver_interface=127.28.83.65
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.83.84:60607/kms with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:03:45.556525 9668 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:03:45.558036 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Bootstrap starting.
I20260810 18:03:45.574429 9668 log.cc:824] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:45.580775 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:03:45.581163 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Bootstrap complete.
I20260810 18:03:45.585376 9668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } }
I20260810 18:03:45.585878 9668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 511d983d3dc5425da8dbd03ad943d05d, State: Initialized, Role: FOLLOWER
I20260810 18:03:45.586341 9668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } }
I20260810 18:03:45.586483 9668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:03:45.586568 9668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:03:45.586707 9668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:03:45.588538 9668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } }
I20260810 18:03:45.588887 9668 leader_election.cc:302] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 511d983d3dc5425da8dbd03ad943d05d; no voters:
I20260810 18:03:45.589197 9668 leader_election.cc:288] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:03:45.589331 9672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:03:45.589594 9672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [term 2 LEADER]: Becoming Leader. State: Replica: 511d983d3dc5425da8dbd03ad943d05d, State: Running, Role: LEADER
I20260810 18:03:45.589963 9672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } }
I20260810 18:03:45.590420 9668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:03:45.591329 9673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "511d983d3dc5425da8dbd03ad943d05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } } }
I20260810 18:03:45.591446 9673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:45.591820 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 511d983d3dc5425da8dbd03ad943d05d. Latest consensus state: current_term: 2 leader_uuid: "511d983d3dc5425da8dbd03ad943d05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "511d983d3dc5425da8dbd03ad943d05d" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40035 } } }
I20260810 18:03:45.591954 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:45.592252 9681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:03:45.592644 9668 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 18:03:45.598502 9681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:03:45.599566 9681 catalog_manager.cc:1259] Loaded cluster ID: 38ed272f42cf4d30a94ccf21ada01303
I20260810 18:03:45.599686 9681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:03:45.602459 9681 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:03:45.603184 9681 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Loaded TSK: 0
I20260810 18:03:45.604465 9681 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:03:45.684957 9670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:45.685214 9670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:45.685325 9670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:45.690212 9670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:45.690343 9670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:45.690469 9670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:03:45.696098 9670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.83.84:60607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:32971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=42143
--tserver_master_addrs=127.28.83.126:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:45.697515 9670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:45.698894 9670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:45.711360 9706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:45.711306 9709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:45.711797 9707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:45.711953 9670 server_base.cc:1034] running on GCE node
I20260810 18:03:45.712520 9670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:45.713258 9670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:45.714437 9670 hybrid_clock.cc:648] HybridClock initialized: now 1786385025714271 us; error 184 us; skew 500 ppm
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:45.717820 9670 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:03:45.718926 9670 webserver.cc:460] Webserver started at http://127.28.83.65:42143/ using document root <none> and password file <none>
I20260810 18:03:45.719656 9670 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:45.719767 9670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:45.748996 9670 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.002s sys 0.000s
I20260810 18:03:45.752368 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:45.753942 9670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:03:45.754073 9670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "01d4cde497364fe48725c68c243fc250"
format_stamp: "Formatted at 2026-08-10 18:03:44 on dist-test-slave-srgv"
server_key: "574ba20878fa2604c92f3963ea208e8a"
server_key_iv: "d5d08ae9d17ac133f4af8be9bec866e0"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:45.754813 9670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:45.774214 9670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:45.777848 9670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:45.778057 9670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:45.778745 9670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:45.779860 9670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:45.779927 9670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:45.779987 9670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:45.780074 9670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 18:03:45.791299 9670 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:32971
I20260810 18:03:45.792739 9670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:03:45.797565 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 9670
I20260810 18:03:45.798010 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:60607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:44763
--local_ip_for_outbound_sockets=127.28.83.66
--tserver_master_addrs=127.28.83.126:40035
--webserver_port=39631
--webserver_interface=127.28.83.66
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.83.84:60607/kms with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:03:45.801206 9831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:32971 every 8 connection(s)
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:45.821625 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:45.803567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40035 (local address 127.28.83.65:39211)
0810 18:03:45.804119 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:45.804172 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:45.805267 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:45.806722 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:45.806729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:45.806983 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:45.807431 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:45.807443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:45.809530 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:45.809535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:45.810926 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:45.810937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:45.812212 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:45.812217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:45.812399 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:45.813109 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:45.813131 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:45.814948 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:45.817810 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:45.817816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:45.817831 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:45.818194 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:45.818669 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:45.818672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:45.818673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:45.818848 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:45.819677 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:45.819684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:45.820001 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 18:03:45.820224 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 18:03:45.820560 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":147,"threads_started":1}
I20260810 18:03:45.821615 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:45.804968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:39211 (local address 127.28.83.126:40035)
0810 18:03:45.805509 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:45.805531 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:45.805579 (+ 48us) server_negotiation.cc:407] Connection header received
0810 18:03:45.805856 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:45.805873 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:45.806280 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:45.806662 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:45.807611 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:45.809359 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:45.811078 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:45.811792 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:45.811957 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:45.815263 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:45.815270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:45.815287 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:45.815321 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:45.817658 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:45.818371 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:45.818377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:45.818384 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:45.818547 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:45.818964 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:45.818967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:45.818969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:45.819482 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:45.819615 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:45.820427 (+ 812us) server_negotiation.cc:299] Negotiation successful
0810 18:03:45.820691 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":146,"threads_started":1}
I20260810 18:03:45.828847 9832 heartbeater.cc:347] Connected to a master server at 127.28.83.126:40035
I20260810 18:03:45.829231 9832 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:45.833626 9832 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
W20260810 18:03:45.836045 9832 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:40035 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 18:03:45.836303 9832 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:45.836548 9832 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
I20260810 18:03:45.837520 9832 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:45.837765 9832 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
W20260810 18:03:45.838564 9832 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:03:45.966802 9834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:45.967222 9834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:45.967355 9834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:45.973508 9834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:45.973596 9834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:45.973698 9834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:03:45.981695 9834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.83.84:60607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:44763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=39631
--tserver_master_addrs=127.28.83.126:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:45.983067 9834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:45.984457 9834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:45.997578 9844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:45.997742 9842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:45.997578 9841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:45.998561 9834 server_base.cc:1034] running on GCE node
I20260810 18:03:45.999075 9834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:45.999871 9834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:46.001070 9834 hybrid_clock.cc:648] HybridClock initialized: now 1786385026001044 us; error 42 us; skew 500 ppm
Aug 10 18:03:45 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:46.004032 9834 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:03:46.004911 9834 webserver.cc:460] Webserver started at http://127.28.83.66:39631/ using document root <none> and password file <none>
I20260810 18:03:46.005573 9834 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:46.005681 9834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:03:46 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:46.026196 9834 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:03:46.029385 9852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:46.030954 9834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:03:46.031101 9834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "ec93b5531fa74d33bf5370ce9240cdc5"
format_stamp: "Formatted at 2026-08-10 18:03:44 on dist-test-slave-srgv"
server_key: "65928ecca3f42633cb6cf7d6c1848dfb"
server_key_iv: "bc50b90647a24c4cd226253ac473856a"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:46.031829 9834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:46.054504 9834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:46.058344 9834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:46.058620 9834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:46.059415 9834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:46.060722 9834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:46.060818 9834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:46.060906 9834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:46.060962 9834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:46.072403 9834 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:44763
I20260810 18:03:46.073846 9834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:03:46.078891 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:44763 every 8 connection(s)
I20260810 18:03:46.082885 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 9834
I20260810 18:03:46.083392 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.83.84:60607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:40051
--local_ip_for_outbound_sockets=127.28.83.67
--tserver_master_addrs=127.28.83.126:40035
--webserver_port=43165
--webserver_interface=127.28.83.67
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.83.84:60607/kms with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:03:46 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:46.100692 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:46.080737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:51869 (local address 127.28.83.126:40035)
0810 18:03:46.080876 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:46.080880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:46.082191 (+ 1311us) server_negotiation.cc:407] Connection header received
0810 18:03:46.087128 (+ 4937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:46.087135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:46.087212 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:46.087312 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:46.089059 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.090029 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.092074 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.092549 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.092665 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:46.096267 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:46.096273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:46.096279 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:46.096317 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:46.098230 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:46.098883 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:46.098886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:46.098889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:46.098976 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:46.099368 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:46.099372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:46.099373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:46.099596 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:46.099680 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:46.100351 (+ 671us) server_negotiation.cc:299] Negotiation successful
0810 18:03:46.100492 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:03:46.102540 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:46.081289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40035 (local address 127.28.83.66:51869)
0810 18:03:46.082012 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:46.082072 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:46.086791 (+ 4719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:46.087642 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:46.087666 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:46.088065 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:46.088844 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:46.088869 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.090218 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.090224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:46.091907 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:46.091922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.092914 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.092918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:46.093173 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:46.094003 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:46.094029 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:46.096038 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:46.098373 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:46.098379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:46.098396 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:46.098755 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:46.099086 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:46.099089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:46.099091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:46.099262 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:46.099763 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:46.099769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:46.100076 (+ 307us) client_negotiation.cc:769] Sending connection context
0810 18:03:46.101154 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0810 18:03:46.101523 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":169,"threads_started":1}
I20260810 18:03:46.104015 9970 heartbeater.cc:347] Connected to a master server at 127.28.83.126:40035
I20260810 18:03:46.104305 9970 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:46.104750 9970 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
W20260810 18:03:46.106518 9970 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:40035 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 18:03:46.106639 9970 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:46.106827 9970 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
I20260810 18:03:46.107631 9970 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:46.107811 9970 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
W20260810 18:03:46.108556 9970 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:03:46.247526 9973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:46.247814 9973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:46.247925 9973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:46.252417 9973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:46.252492 9973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:46.252604 9973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:03:46.258432 9973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.83.84:60607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:40051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=43165
--tserver_master_addrs=127.28.83.126:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:46.259857 9973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:46.261312 9973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:46.272965 9982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:46.272991 9980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:46.273564 9973 server_base.cc:1034] running on GCE node
W20260810 18:03:46.274151 9979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:46.274696 9973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:46.275507 9973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:46.276692 9973 hybrid_clock.cc:648] HybridClock initialized: now 1786385026276661 us; error 46 us; skew 500 ppm
Aug 10 18:03:46 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:46.279575 9973 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:03:46.280571 9973 webserver.cc:460] Webserver started at http://127.28.83.67:43165/ using document root <none> and password file <none>
I20260810 18:03:46.281213 9973 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:46.281319 9973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:03:46 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:46.304967 9973 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260810 18:03:46.308293 9990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:46.309862 9973 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 18:03:46.309983 9973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "1b508c31280c4113b3e1afea0998a02a"
format_stamp: "Formatted at 2026-08-10 18:03:45 on dist-test-slave-srgv"
server_key: "cbd994c030210bfc895baa2afa1b1983"
server_key_iv: "d69459c75b3dc072556fea8a6a6479f3"
server_key_version: "kuduclusterkey@0"
I20260810 18:03:46.310801 9973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:46.332198 9973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:46.335902 9973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:46.336172 9973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:46.337095 9973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:46.338409 9973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:46.338524 9973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:46.338699 9973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:46.338788 9973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:46.353655 9973 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:40051
I20260810 18:03:46.355095 9973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:03:46.356449 10107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:40051 every 8 connection(s)
I20260810 18:03:46.362589 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 9973
Aug 10 18:03:46 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:46.389429 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:46.358722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40035 (local address 127.28.83.67:49007)
0810 18:03:46.365743 (+ 7021us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:46.365805 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:46.366959 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:46.367773 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:46.367783 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:46.368130 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:46.368788 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:46.368806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.370017 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.370020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:46.371400 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:46.371415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.374217 (+ 2802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.374221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:46.374415 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:46.381591 (+ 7176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:46.381613 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:46.383683 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:46.385955 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:46.385961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:46.385978 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:46.386367 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:46.386742 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:46.386744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:46.386746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:46.386928 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:46.387406 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:46.387412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:46.387740 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 18:03:46.387942 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 18:03:46.388291 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6763,"thread_start_us":137,"threads_started":1}
I20260810 18:03:46.390291 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:46.359956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:49007 (local address 127.28.83.126:40035)
0810 18:03:46.360087 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:46.360092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:46.365928 (+ 5836us) server_negotiation.cc:407] Connection header received
0810 18:03:46.367219 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:46.367224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:46.367282 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:46.367337 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:46.368962 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.369836 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.373526 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.373912 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.373998 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:46.383896 (+ 9898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:46.383899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:46.383901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:46.383950 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:46.385844 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:46.386520 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:46.386525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:46.386529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:46.386637 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:46.387036 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:46.387039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:46.387041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:46.387267 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:46.387352 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:46.390014 (+ 2662us) server_negotiation.cc:299] Negotiation successful
0810 18:03:46.390142 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:03:46.390654 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:46.363340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.28.83.126:40035)
0810 18:03:46.376889 (+ 13549us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:46.376896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:46.376912 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:03:46.376955 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:46.376960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:46.377015 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:46.377135 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:46.378388 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.379340 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.380791 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:46.381240 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:46.381348 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:46.383273 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:46.383276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:46.383281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:46.383314 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:46.387047 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:46.388299 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:46.388301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:46.388303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:46.388378 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:46.388668 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:46.388671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:46.388673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:46.388848 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:46.389043 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:46.390363 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0810 18:03:46.390488 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13461,"thread_start_us":98,"threads_started":1}
I20260810 18:03:46.391938 10108 heartbeater.cc:347] Connected to a master server at 127.28.83.126:40035
I20260810 18:03:46.392176 10108 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:46.392709 10108 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
W20260810 18:03:46.394399 10108 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:40035 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 18:03:46.394533 10108 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:46.394699 10108 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
I20260810 18:03:46.395543 10108 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:46.395727 10108 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
W20260810 18:03:46.398490 10108 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:03:46.838876 9832 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:46.839197 9832 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
I20260810 18:03:46.840412 9631 ts_manager.cc:194] Registered new tserver with Master: 01d4cde497364fe48725c68c243fc250 (127.28.83.65:32971)
I20260810 18:03:46.842157 9631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:39211
Aug 10 18:03:46 dist-test-slave-srgv krb5kdc[8629](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:47.108824 9970 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:47.109126 9970 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
I20260810 18:03:47.109884 9631 ts_manager.cc:194] Registered new tserver with Master: ec93b5531fa74d33bf5370ce9240cdc5 (127.28.83.66:44763)
I20260810 18:03:47.110667 9631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:51869
Aug 10 18:03:47 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
I20260810 18:03:47.398732 10108 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:47.399017 10108 heartbeater.cc:511] Master 127.28.83.126:40035 requested a full tablet report, sending...
I20260810 18:03:47.399775 9631 ts_manager.cc:194] Registered new tserver with Master: 1b508c31280c4113b3e1afea0998a02a (127.28.83.67:40051)
I20260810 18:03:47.400583 9631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:49007
I20260810 18:03:47.408969 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:03:47.422118 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.410925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.28.83.126:40035)
0810 18:03:47.411537 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:47.411542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:47.411897 (+ 355us) server_negotiation.cc:407] Connection header received
0810 18:03:47.411950 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:47.411954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:47.412003 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:47.412092 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:47.413077 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.413961 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.415246 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.415680 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.415794 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.417725 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:47.417729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:47.417732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:47.417760 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:47.419742 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:47.420269 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:47.420272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:47.420274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:47.420360 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:47.420627 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:47.420630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:47.420632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:47.420878 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:47.421025 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:47.421775 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 18:03:47.421908 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":86,"threads_started":1}
I20260810 18:03:47.426785 9631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:03:47.432312 9631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:03:47.458863 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.446916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:40051 (local address 127.0.0.1:40914)
0810 18:03:47.450382 (+ 3466us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:47.450439 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:47.450576 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:47.452824 (+ 2248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:47.452830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:47.452851 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:47.452977 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.452983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.455122 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.455125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.456498 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.456510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.458300 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.458304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.458420 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.458463 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 18:03:47.458575 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 18:03:47.458711 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3290,"thread_start_us":98,"threads_started":1}
I20260810 18:03:47.459906 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.447721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.28.83.67:40051)
0810 18:03:47.451807 (+ 4086us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:47.451814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:47.451839 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:03:47.452281 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:47.452287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:47.452505 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:47.452654 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:47.453102 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.454994 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.456669 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.457485 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.459454 (+ 1969us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.459623 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:47.459674 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 18:03:47.459765 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3973,"thread_start_us":57,"threads_started":1}
I20260810 18:03:47.463366 10038 tablet_service.cc:1467] Processing CreateTablet for tablet 6913766cc9c8498292f3f6565909114b (DEFAULT_TABLE table=test-table [id=55734a18d377458ba990b873c963a14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:03:47.464543 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6913766cc9c8498292f3f6565909114b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:47.466168 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.449789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:32971 (local address 127.0.0.1:60548)
0810 18:03:47.451976 (+ 2187us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:47.452005 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:47.452085 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:47.460968 (+ 8883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:47.460972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:47.460999 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:47.461110 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.461116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.463515 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.463518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.464635 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.464647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.465754 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.465758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.465851 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.465877 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:03:47.465939 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:03:47.466004 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2123,"thread_start_us":103,"threads_started":1}
I20260810 18:03:47.467207 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.449949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.28.83.65:32971)
0810 18:03:47.460479 (+ 10530us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:47.460487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:47.460512 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:03:47.460588 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:47.460593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:47.460797 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:47.461270 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:47.461508 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.462594 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.464796 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.465604 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.466764 (+ 1160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.466914 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:47.466966 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 18:03:47.467069 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10407,"thread_start_us":96,"threads_started":1}
I20260810 18:03:47.470465 9755 tablet_service.cc:1467] Processing CreateTablet for tablet 6913766cc9c8498292f3f6565909114b (DEFAULT_TABLE table=test-table [id=55734a18d377458ba990b873c963a14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:03:47.471582 9755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6913766cc9c8498292f3f6565909114b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:47.476982 10142 tablet_bootstrap.cc:492] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: Bootstrap starting.
I20260810 18:03:47.480391 10142 tablet_bootstrap.cc:654] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:47.481443 10142 log.cc:824] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:47.483534 10143 tablet_bootstrap.cc:492] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250: Bootstrap starting.
I20260810 18:03:47.485865 10143 tablet_bootstrap.cc:654] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:47.487627 10143 log.cc:824] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:47.490429 10142 tablet_bootstrap.cc:492] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: No bootstrap required, opened a new log
I20260810 18:03:47.490720 10142 ts_tablet_manager.cc:1397] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260810 18:03:47.491183 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.449093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:44763 (local address 127.0.0.1:50042)
0810 18:03:47.451418 (+ 2325us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:47.451430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:47.451512 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:47.469500 (+ 17988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:47.469504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:47.469527 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:47.469659 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.469665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.482115 (+ 12450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.482120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.483256 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.483270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.490666 (+ 7396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.490670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.490805 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.490840 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 18:03:47.490909 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:03:47.491017 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2265,"thread_start_us":105,"threads_started":1}
I20260810 18:03:47.491858 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.449265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.28.83.66:44763)
0810 18:03:47.462892 (+ 13627us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:47.462898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:47.462921 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:03:47.462995 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:47.462999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:47.463191 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:47.463331 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:47.480623 (+ 17292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.481983 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.489489 (+ 7506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.490497 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.491418 (+ 921us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.491581 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:47.491635 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 18:03:47.491725 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13513,"thread_start_us":102,"threads_started":1}
I20260810 18:03:47.494881 9882 tablet_service.cc:1467] Processing CreateTablet for tablet 6913766cc9c8498292f3f6565909114b (DEFAULT_TABLE table=test-table [id=55734a18d377458ba990b873c963a14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:03:47.495920 9882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6913766cc9c8498292f3f6565909114b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:47.508494 10142 raft_consensus.cc:348] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } }
I20260810 18:03:47.509014 10142 raft_consensus.cc:374] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:47.509135 10142 raft_consensus.cc:729] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b508c31280c4113b3e1afea0998a02a, State: Initialized, Role: FOLLOWER
I20260810 18:03:47.510735 10142 consensus_queue.cc:260] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } }
I20260810 18:03:47.512028 10142 ts_tablet_manager.cc:1428] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260810 18:03:47.512454 10108 heartbeater.cc:503] Master 127.28.83.126:40035 was elected leader, sending a full tablet report...
I20260810 18:03:47.516304 10146 tablet_bootstrap.cc:492] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: Bootstrap starting.
I20260810 18:03:47.519016 10146 tablet_bootstrap.cc:654] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:47.523180 10143 tablet_bootstrap.cc:492] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250: No bootstrap required, opened a new log
I20260810 18:03:47.523453 10143 ts_tablet_manager.cc:1397] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20260810 18:03:47.528519 10143 raft_consensus.cc:348] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } }
I20260810 18:03:47.528964 10143 raft_consensus.cc:374] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:47.529188 10143 raft_consensus.cc:729] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d4cde497364fe48725c68c243fc250, State: Initialized, Role: FOLLOWER
I20260810 18:03:47.529752 10143 consensus_queue.cc:260] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } }
I20260810 18:03:47.530927 10143 ts_tablet_manager.cc:1428] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:03:47.532480 10146 log.cc:824] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:47.532932 9832 heartbeater.cc:503] Master 127.28.83.126:40035 was elected leader, sending a full tablet report...
I20260810 18:03:47.539151 10146 tablet_bootstrap.cc:492] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: No bootstrap required, opened a new log
I20260810 18:03:47.539418 10146 ts_tablet_manager.cc:1397] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20260810 18:03:47.543426 10146 raft_consensus.cc:348] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } }
I20260810 18:03:47.543788 10146 raft_consensus.cc:374] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:47.543900 10146 raft_consensus.cc:729] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec93b5531fa74d33bf5370ce9240cdc5, State: Initialized, Role: FOLLOWER
I20260810 18:03:47.545601 10146 consensus_queue.cc:260] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } }
I20260810 18:03:47.546769 10146 ts_tablet_manager.cc:1428] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:03:47.547191 9970 heartbeater.cc:503] Master 127.28.83.126:40035 was elected leader, sending a full tablet report...
W20260810 18:03:47.561132 9833 tablet.cc:2369] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:03:47.583853 9971 tablet.cc:2369] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 18:03:47 dist-test-slave-srgv krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for HTTP/127.28.83.84@KRBTEST.COM
W20260810 18:03:47.611613 10109 tablet.cc:2369] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:03:47.622895 10151 raft_consensus.cc:482] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:03:47.623142 10151 raft_consensus.cc:504] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } }
I20260810 18:03:47.624655 10151 leader_election.cc:288] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b508c31280c4113b3e1afea0998a02a (127.28.83.67:40051), 01d4cde497364fe48725c68c243fc250 (127.28.83.65:32971)
I20260810 18:03:47.636518 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.625276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:38537 (local address 127.28.83.67:40051)
0810 18:03:47.625428 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:47.625433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:47.627045 (+ 1612us) server_negotiation.cc:407] Connection header received
0810 18:03:47.628141 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:47.628149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:47.628217 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:47.628268 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:47.629512 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.630259 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.633860 (+ 3601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.634500 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.634600 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.634654 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:47.636238 (+ 1584us) server_negotiation.cc:299] Negotiation successful
0810 18:03:47.636360 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:03:47.636981 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.625332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:57317 (local address 127.28.83.65:32971)
0810 18:03:47.625479 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:47.625483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:47.629877 (+ 4394us) server_negotiation.cc:407] Connection header received
0810 18:03:47.630052 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:47.630055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:47.630100 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:47.630144 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:47.631088 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.631846 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.634793 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.635414 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.635506 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.635550 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:47.636741 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0810 18:03:47.636857 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:03:47.637329 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.624826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:40051 (local address 127.28.83.66:38537)
0810 18:03:47.626944 (+ 2118us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:47.626961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:47.627925 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:47.628644 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:47.628647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:47.628668 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:47.628816 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.628825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.630556 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.630569 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.633725 (+ 3156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.633736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.634968 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.634971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.635068 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.635091 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:03:47.637036 (+ 1945us) client_negotiation.cc:240] Negotiation successful
0810 18:03:47.637122 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2042,"mutex_wait_us":1,"thread_start_us":130,"threads_started":1}
I20260810 18:03:47.638027 10043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913766cc9c8498292f3f6565909114b" candidate_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b508c31280c4113b3e1afea0998a02a" is_pre_election: true
I20260810 18:03:47.638401 10043 raft_consensus.cc:2454] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec93b5531fa74d33bf5370ce9240cdc5 in term 0.
I20260810 18:03:47.638628 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.624954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:32971 (local address 127.28.83.66:57317)
0810 18:03:47.629769 (+ 4815us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:47.629783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:47.629943 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:47.630265 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:47.630267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:47.630284 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:47.630595 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.630603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.632311 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.632314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.634384 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:47.634394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.636134 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.636141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:47.636331 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.636356 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:03:47.638098 (+ 1742us) client_negotiation.cc:240] Negotiation successful
0810 18:03:47.638283 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4744,"mutex_wait_us":614,"thread_start_us":129,"threads_started":1}
I20260810 18:03:47.638978 9853 leader_election.cc:302] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b508c31280c4113b3e1afea0998a02a, ec93b5531fa74d33bf5370ce9240cdc5; no voters:
I20260810 18:03:47.639279 10151 raft_consensus.cc:2781] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:03:47.639369 10151 raft_consensus.cc:482] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:03:47.639429 10151 raft_consensus.cc:3037] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:47.639559 9785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913766cc9c8498292f3f6565909114b" candidate_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d4cde497364fe48725c68c243fc250" is_pre_election: true
I20260810 18:03:47.639834 9785 raft_consensus.cc:2454] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec93b5531fa74d33bf5370ce9240cdc5 in term 0.
I20260810 18:03:47.640964 10151 raft_consensus.cc:504] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } }
I20260810 18:03:47.641343 10151 leader_election.cc:288] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [CANDIDATE]: Term 1 election: Requested vote from peers 1b508c31280c4113b3e1afea0998a02a (127.28.83.67:40051), 01d4cde497364fe48725c68c243fc250 (127.28.83.65:32971)
I20260810 18:03:47.641911 9785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913766cc9c8498292f3f6565909114b" candidate_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01d4cde497364fe48725c68c243fc250"
I20260810 18:03:47.642032 9785 raft_consensus.cc:3037] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:47.643332 9785 raft_consensus.cc:2454] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec93b5531fa74d33bf5370ce9240cdc5 in term 1.
I20260810 18:03:47.644249 9854 leader_election.cc:302] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d4cde497364fe48725c68c243fc250, ec93b5531fa74d33bf5370ce9240cdc5; no voters:
I20260810 18:03:47.644610 10151 raft_consensus.cc:2781] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:03:47.644860 10151 raft_consensus.cc:686] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 1 LEADER]: Becoming Leader. State: Replica: ec93b5531fa74d33bf5370ce9240cdc5, State: Running, Role: LEADER
I20260810 18:03:47.644826 10043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6913766cc9c8498292f3f6565909114b" candidate_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b508c31280c4113b3e1afea0998a02a"
I20260810 18:03:47.644954 10043 raft_consensus.cc:3037] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:47.645226 10151 consensus_queue.cc:237] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } }
I20260810 18:03:47.646283 10043 raft_consensus.cc:2454] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec93b5531fa74d33bf5370ce9240cdc5 in term 1.
I20260810 18:03:47.648581 9631 catalog_manager.cc:5495] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec93b5531fa74d33bf5370ce9240cdc5 (127.28.83.66). New cstate: current_term: 1 leader_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec93b5531fa74d33bf5370ce9240cdc5" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 } health_report { overall_health: UNKNOWN } } }
I20260810 18:03:47.700618 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.695426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.28.83.66:44763)
0810 18:03:47.695553 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:47.695557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:47.695573 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:03:47.695669 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:47.695672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:47.695756 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:47.695888 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:03:47.696302 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.697112 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.698363 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:47.698684 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:47.698809 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:47.699257 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:03:47.699837 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:03:47.699961 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:47.700405 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 18:03:47.700470 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:03:47.708194 9785 raft_consensus.cc:1261] T 6913766cc9c8498292f3f6565909114b P 01d4cde497364fe48725c68c243fc250 [term 1 FOLLOWER]: Refusing update from remote peer ec93b5531fa74d33bf5370ce9240cdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:03:47.708194 10043 raft_consensus.cc:1261] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [term 1 FOLLOWER]: Refusing update from remote peer ec93b5531fa74d33bf5370ce9240cdc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:03:47.709123 10156 consensus_queue.cc:1035] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b508c31280c4113b3e1afea0998a02a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:03:47.709295 10151 consensus_queue.cc:1035] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d4cde497364fe48725c68c243fc250" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 32971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:03:47.735339 10159 mvcc.cc:205] Tried to move back new op lower bound from 7317033073483964416 to 7317033073238224896. Current Snapshot: MvccSnapshot[applied={T|T < 7317033073483964416}]
I20260810 18:03:47.740495 10160 mvcc.cc:205] Tried to move back new op lower bound from 7317033073483964416 to 7317033073238224896. Current Snapshot: MvccSnapshot[applied={T|T < 7317033073483964416}]
I20260810 18:03:47.744962 10161 mvcc.cc:205] Tried to move back new op lower bound from 7317033073483964416 to 7317033073238224896. Current Snapshot: MvccSnapshot[applied={T|T < 7317033073483964416}]
I20260810 18:03:47.751843 9631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:03:47.752024 9631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:03:47.754941 9631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 511d983d3dc5425da8dbd03ad943d05d: Sending DeleteTablet for 3 replicas of tablet 6913766cc9c8498292f3f6565909114b
I20260810 18:03:47.755666 10038 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913766cc9c8498292f3f6565909114b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:03:47 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:40914
I20260810 18:03:47.756683 9882 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913766cc9c8498292f3f6565909114b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:03:47 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:50042
I20260810 18:03:47.757643 9755 tablet_service.cc:1514] Processing DeleteTablet for tablet 6913766cc9c8498292f3f6565909114b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:03:47 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:60548
I20260810 18:03:47.759043 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 9670
I20260810 18:03:47.761631 10168 tablet_replica.cc:330] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: stopping tablet replica
I20260810 18:03:47.761982 10168 raft_consensus.cc:2229] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:03:47.762329 10168 raft_consensus.cc:2258] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:03:47.764009 10168 ts_tablet_manager.cc:1905] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:03:47.769601 10169 tablet_replica.cc:330] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: stopping tablet replica
I20260810 18:03:47.769914 10169 raft_consensus.cc:2229] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:03:47.770344 10169 raft_consensus.cc:2258] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:03:47.777860 10169 ts_tablet_manager.cc:1905] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:03:47.777928 10168 ts_tablet_manager.cc:1918] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:03:47.778039 10168 log.cc:1196] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/6913766cc9c8498292f3f6565909114b
I20260810 18:03:47.778422 10168 ts_tablet_manager.cc:1939] T 6913766cc9c8498292f3f6565909114b P 1b508c31280c4113b3e1afea0998a02a: Deleting consensus metadata
I20260810 18:03:47.779884 9616 catalog_manager.cc:4841] TS 1b508c31280c4113b3e1afea0998a02a (127.28.83.67:40051): tablet 6913766cc9c8498292f3f6565909114b (table test-table [id=55734a18d377458ba990b873c963a14e]) successfully deleted
I20260810 18:03:47.780815 10169 ts_tablet_manager.cc:1918] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:03:47.780915 10169 log.cc:1196] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/6913766cc9c8498292f3f6565909114b
I20260810 18:03:47.781256 10169 ts_tablet_manager.cc:1939] T 6913766cc9c8498292f3f6565909114b P ec93b5531fa74d33bf5370ce9240cdc5: Deleting consensus metadata
I20260810 18:03:47.782402 9618 catalog_manager.cc:4841] TS ec93b5531fa74d33bf5370ce9240cdc5 (127.28.83.66:44763): tablet 6913766cc9c8498292f3f6565909114b (table test-table [id=55734a18d377458ba990b873c963a14e]) successfully deleted
W20260810 18:03:47.782860 9617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:32971 (error 108)
I20260810 18:03:47.783839 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 9834
I20260810 18:03:47.795841 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 9973
I20260810 18:03:47.798731 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:47.798364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:32971 (local address 127.0.0.1:60564)
0810 18:03:47.798563 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:47.798635 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:32971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260810 18:03:47.801972 9617 catalog_manager.cc:4584] TS 01d4cde497364fe48725c68c243fc250 (127.28.83.65:32971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6913766cc9c8498292f3f6565909114b: Network error: Client connection negotiation failed: client connection to 127.28.83.65:32971: connect: Connection refused (error 111)
I20260810 18:03:47.807335 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 9599
I20260810 18:03:47.816824 29005 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 18:03:48.161198 29005 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 18:03:48.161314 29005 mini_ranger.cc:66] Stopping Ranger...
I20260810 18:03:48.505868 29005 mini_ranger.cc:68] Stopped Ranger
2026-08-10 18:03:48.506 UTC [8659] LOG: received smart shutdown request
2026-08-10 18:03:48.510 UTC [8659] LOG: background worker "logical replication launcher" (PID 8668) exited with exit code 1
2026-08-10 18:03:48.510 UTC [8663] LOG: shutting down
2026-08-10 18:03:48.686 UTC [8659] LOG: database system is shut down
2026-08-10T18:03:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49181 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:03:48 dist-test-slave-srgv krb5kdc[10181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:03:48 dist-test-slave-srgv krb5kdc[10181](info): set up 2 sockets
Aug 10 18:03:48 dist-test-slave-srgv krb5kdc[10181](info): commencing operation
krb5kdc: starting...
W20260810 18:03:49.873148 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:03:49 dist-test-slave-srgv krb5kdc[10181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:03:49.999615 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40389
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:40389,127.28.83.125:35271,127.28.83.124:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:50.111483 10194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:50.111807 10194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:50.111893 10194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:50.115841 10194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:03:50.115952 10194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:50.115993 10194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:50.116031 10194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:03:50.116062 10194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:03:50.121307 10194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:40389,127.28.83.125:35271,127.28.83.124:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:50.122424 10194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:50.123454 10194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:50.131772 10199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:50.131804 10202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:50.131979 10194 server_base.cc:1034] running on GCE node
W20260810 18:03:50.132144 10200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:50.132737 10194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:50.133957 10194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:50.135129 10194 hybrid_clock.cc:648] HybridClock initialized: now 1786385030135102 us; error 40 us; skew 500 ppm
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:50.138020 10194 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:03:50.138934 10194 webserver.cc:460] Webserver started at http://127.28.83.126:36795/ using document root <none> and password file <none>
I20260810 18:03:50.139506 10194 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:50.139603 10194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:50.139846 10194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:03:50.141903 10194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "800bbce541c34bd38b56ce4043d5df44"
format_stamp: "Formatted at 2026-08-10 18:03:50 on dist-test-slave-srgv"
server_key: "5256d9a6e58f95ad1c841ffd1fc07d58"
server_key_iv: "d680c4087554b0bc105e78069855b054"
server_key_version: "encryptionkey@0"
I20260810 18:03:50.142472 10194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "800bbce541c34bd38b56ce4043d5df44"
format_stamp: "Formatted at 2026-08-10 18:03:50 on dist-test-slave-srgv"
server_key: "5256d9a6e58f95ad1c841ffd1fc07d58"
server_key_iv: "d680c4087554b0bc105e78069855b054"
server_key_version: "encryptionkey@0"
I20260810 18:03:50.146323 10194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:03:50.148734 10209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:50.150009 10194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:03:50.150183 10194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "800bbce541c34bd38b56ce4043d5df44"
format_stamp: "Formatted at 2026-08-10 18:03:50 on dist-test-slave-srgv"
server_key: "5256d9a6e58f95ad1c841ffd1fc07d58"
server_key_iv: "d680c4087554b0bc105e78069855b054"
server_key_version: "encryptionkey@0"
I20260810 18:03:50.150303 10194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:50.163645 10194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:50.167227 10194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:50.167445 10194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:50.174551 10194 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:40389
I20260810 18:03:50.174564 10264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:40389 every 8 connection(s)
I20260810 18:03:50.175506 10194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:03:50.175937 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 10194
I20260810 18:03:50.176075 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:03:50.176362 29005 external_mini_cluster.cc:1428] Setting key 787cf38ccfa5bf8736ae35d735ea5772
I20260810 18:03:50.178511 10265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:50.182634 10265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.83.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.125@KRBTEST.COM" created.
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:50.198891 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.185561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.28.83.126:40389)
0810 18:03:50.185990 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.186010 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.186278 (+ 268us) server_negotiation.cc:407] Connection header received
0810 18:03:50.187362 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.187366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.187541 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.187734 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.188558 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.189555 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.190617 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.190993 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.191172 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.193882 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.193891 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.193908 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.193941 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.195670 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.196164 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.196170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.196172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.196279 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.196609 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.196612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.196614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.197005 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.197151 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.197578 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.197832 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":282,"thread_start_us":142,"threads_started":1}
I20260810 18:03:50.198896 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.185562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40389 (local address 127.0.0.1:59618)
0810 18:03:50.186136 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.186171 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:50.187051 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:50.187728 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:50.187734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:50.188164 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:50.188410 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.188423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.189678 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.189681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.190488 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.190499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.191153 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.191157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.191261 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.191915 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:50.191945 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:50.193714 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:50.195802 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.195812 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.195821 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.196042 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.196384 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.196387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.196389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.196499 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.197140 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:50.197147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:50.197276 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 18:03:50.197497 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 18:03:50.197889 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":110,"spinlock_wait_cycles":40832,"thread_start_us":164,"threads_started":1}
I20260810 18:03:50.201356 10265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:03:50.202178 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.201842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:35271 (local address 127.0.0.1:60030)
0810 18:03:50.201990 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.202094 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 18:03:50.202353 10213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
I20260810 18:03:50.203284 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.203061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:35271 (local address 127.0.0.1:60036)
0810 18:03:50.203173 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.203219 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:03:50.203456 10265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.125:35271: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab.
Entry for principal kudu/127.28.83.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab.
I20260810 18:03:50.207943 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.125:35271
--webserver_interface=127.28.83.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab
--principal=kudu/127.28.83.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:40389,127.28.83.125:35271,127.28.83.124:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:03:50.252730 10265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } attempt: 1
I20260810 18:03:50.253667 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.253337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:35271 (local address 127.0.0.1:60050)
0810 18:03:50.253517 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.253588 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 18:03:50.254637 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.254292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:35271 (local address 127.0.0.1:60052)
0810 18:03:50.254516 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.254575 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260810 18:03:50.254863 10265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.125:35271: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
W20260810 18:03:50.319873 10272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:50.320143 10272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:50.320183 10272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 18:03:50.323045 10265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } attempt: 2
I20260810 18:03:50.323880 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.323564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:35271 (local address 127.0.0.1:60056)
0810 18:03:50.323735 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.323805 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 18:03:50.324654 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.324459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:35271 (local address 127.0.0.1:60072)
0810 18:03:50.324557 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.324604 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 18:03:50.324649 10272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:03:50.324719 10272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:50.324764 10272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:50.324800 10272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:03:50.324828 10272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:03:50.324810 10265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.125:35271: Network error: Client connection negotiation failed: client connection to 127.28.83.125:35271: connect: Connection refused (error 111)
I20260810 18:03:50.330093 10272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:40389,127.28.83.125:35271,127.28.83.124:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab
--principal=kudu/127.28.83.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.125:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.83.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:50.331084 10272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:50.332046 10272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:50.340503 10277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:50.340586 10272 server_base.cc:1034] running on GCE node
W20260810 18:03:50.340534 10278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:50.340535 10280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:50.341236 10272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:50.342411 10272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:50.343631 10272 hybrid_clock.cc:648] HybridClock initialized: now 1786385030343618 us; error 31 us; skew 500 ppm
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:50.346576 10272 init.cc:377] Logged in from keytab as kudu/127.28.83.125@KRBTEST.COM (short username kudu)
I20260810 18:03:50.347417 10272 webserver.cc:460] Webserver started at http://127.28.83.125:43623/ using document root <none> and password file <none>
I20260810 18:03:50.347963 10272 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:50.348055 10272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:50.348282 10272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:03:50.350364 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/data/instance:
uuid: "901f8a5b2bb949f7af54b1ee547c1b5b"
format_stamp: "Formatted at 2026-08-10 18:03:50 on dist-test-slave-srgv"
server_key: "32156f8add150f12d0afaeab0e42d419"
server_key_iv: "d123363e4722de6a7a2ff40ccf13e919"
server_key_version: "encryptionkey@0"
I20260810 18:03:50.350962 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/wal/instance:
uuid: "901f8a5b2bb949f7af54b1ee547c1b5b"
format_stamp: "Formatted at 2026-08-10 18:03:50 on dist-test-slave-srgv"
server_key: "32156f8add150f12d0afaeab0e42d419"
server_key_iv: "d123363e4722de6a7a2ff40ccf13e919"
server_key_version: "encryptionkey@0"
I20260810 18:03:50.354854 10272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:03:50.357185 10287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:50.358377 10272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:03:50.358532 10272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/wal
uuid: "901f8a5b2bb949f7af54b1ee547c1b5b"
format_stamp: "Formatted at 2026-08-10 18:03:50 on dist-test-slave-srgv"
server_key: "32156f8add150f12d0afaeab0e42d419"
server_key_iv: "d123363e4722de6a7a2ff40ccf13e919"
server_key_version: "encryptionkey@0"
I20260810 18:03:50.358665 10272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:50.377490 10272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:50.380932 10272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:50.381176 10272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:50.388300 10272 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.125:35271
I20260810 18:03:50.388368 10340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.125:35271 every 8 connection(s)
I20260810 18:03:50.389307 10272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
I20260810 18:03:50.392002 10341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:50.394625 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 10272
I20260810 18:03:50.394768 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-1/wal/instance
I20260810 18:03:50.395030 29005 external_mini_cluster.cc:1428] Setting key 183f45a0f73f2538fa8584812468fe33
I20260810 18:03:50.395852 10341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.83.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.124@KRBTEST.COM" created.
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:50.412097 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.398074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.28.83.126:40389)
0810 18:03:50.398251 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.398256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.399182 (+ 926us) server_negotiation.cc:407] Connection header received
0810 18:03:50.400571 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.400574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.400622 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.400701 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.402403 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.402918 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.404703 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.404934 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.405015 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.408081 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.408084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.408086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.408111 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.409905 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.410432 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.410435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.410437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.410525 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.410886 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.410891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.410892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.411087 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.411230 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.411778 (+ 548us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.411920 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:03:50.413233 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.398431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40389 (local address 127.0.0.1:59620)
0810 18:03:50.398978 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.399040 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:50.400339 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:50.400926 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:50.400938 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:50.401533 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:50.402243 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.402263 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.403065 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.403068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.404595 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.404605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.405066 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.405069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.405255 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.405849 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:50.405864 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:50.407897 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:50.410029 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.410034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.410045 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.410308 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.410625 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.410629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.410631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.410780 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.411244 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:50.411252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:50.411508 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 18:03:50.411718 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 18:03:50.412053 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":117,"threads_started":1}
I20260810 18:03:50.414788 10341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:03:50.415972 10265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } attempt: 3
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab.
Entry for principal kudu/127.28.83.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab.
I20260810 18:03:50.425662 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.124:43279
--webserver_interface=127.28.83.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab
--principal=kudu/127.28.83.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:40389,127.28.83.125:35271,127.28.83.124:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:03:50.426312 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.415321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:35271 (local address 127.0.0.1:60074)
0810 18:03:50.415439 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.415454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:50.415529 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:50.416215 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:50.416218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:50.416404 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:50.416518 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.416529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.417795 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.417798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.418423 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.418432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.419264 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.419267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.419386 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.419895 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:50.419908 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:50.421365 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:50.424350 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.424352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.424353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.424590 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.424891 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.424894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.424896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.425051 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.425668 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:50.425672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:50.425792 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 18:03:50.425912 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 18:03:50.426061 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 18:03:50.427260 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.416344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:35271 (local address 127.0.0.1:60086)
0810 18:03:50.416506 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.416522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:50.416602 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:50.416904 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:50.416906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:50.417060 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:50.417153 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.417158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.418030 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.418036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.419176 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.419186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.419721 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.419725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.419832 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.420310 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:50.420322 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:50.421895 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:50.424275 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.424281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.424286 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.424600 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.425046 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.425051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.425053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.425208 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.426596 (+ 1388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:50.426602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:50.426737 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 18:03:50.426855 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 18:03:50.427024 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 18:03:50.427513 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.416463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.28.83.125:35271)
0810 18:03:50.416715 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.416730 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.416739 (+ 9us) server_negotiation.cc:407] Connection header received
0810 18:03:50.416764 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.416766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.416793 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.416889 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.417310 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.417918 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.419301 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.419600 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.419709 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.422076 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.422081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.422086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.422113 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.424170 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.424724 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.424730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.424735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.424831 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.425364 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.425366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.425368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.425566 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.425691 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.426955 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.427147 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260810 18:03:50.427807 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.415513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.28.83.125:35271)
0810 18:03:50.415841 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.415851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.415876 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:03:50.415907 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.415911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.416047 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.416234 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.416666 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.417669 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.418602 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.419141 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.419268 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.421523 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.421532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.421543 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.421563 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.424238 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.424709 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.424714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.424721 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.424791 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.425183 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.425187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.425189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.425538 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.425691 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.427479 (+ 1788us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.427614 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":237,"thread_start_us":161,"threads_started":1}
I20260810 18:03:50.429369 10341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:03:50.429399 10265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:03:50.430121 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.429868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55194)
0810 18:03:50.429987 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.430070 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:03:50.430353 10291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
I20260810 18:03:50.430418 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.429865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55210)
0810 18:03:50.430312 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.430364 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
I20260810 18:03:50.431188 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.430952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55212)
0810 18:03:50.431081 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.431132 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 18:03:50.431314 10265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43279: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
I20260810 18:03:50.431604 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.431431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55214)
0810 18:03:50.431520 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.431560 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 18:03:50.431802 10341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43279: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
I20260810 18:03:50.462468 10265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } attempt: 1
I20260810 18:03:50.463409 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.463106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55222)
0810 18:03:50.463264 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.463330 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 18:03:50.464152 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.463957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55236)
0810 18:03:50.464062 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.464104 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 18:03:50.464300 10265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43279: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
I20260810 18:03:50.481098 10341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } attempt: 1
I20260810 18:03:50.482038 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.481704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55240)
0810 18:03:50.481879 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.481951 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 18:03:50.482752 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.482535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55254)
0810 18:03:50.482642 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.482695 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:03:50.482885 10341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43279: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
I20260810 18:03:50.539456 10265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } attempt: 2
I20260810 18:03:50.540323 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.540030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55270)
0810 18:03:50.540185 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.540249 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 18:03:50.540998 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.540822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55276)
0810 18:03:50.540912 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.540953 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 18:03:50.540814 10348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:50.541132 10348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:50.541144 10265 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43279: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
W20260810 18:03:50.541211 10348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:50.545295 10348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:03:50.545351 10348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:50.545372 10348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:50.545464 10348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:03:50.545497 10348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:03:50.550827 10348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:40389,127.28.83.125:35271,127.28.83.124:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab
--principal=kudu/127.28.83.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.124:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.83.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:50.551041 10341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } attempt: 2
I20260810 18:03:50.551779 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.551491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55290)
0810 18:03:50.551645 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.551703 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 18:03:50.552103 10348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:50.552513 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.552307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55298)
0810 18:03:50.552421 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.552465 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 18:03:50.552736 10341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43279: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43279: connect: Connection refused (error 111)
I20260810 18:03:50.553359 10348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:50.561677 10354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:50.561685 10357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:50.561700 10355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:50.561903 10348 server_base.cc:1034] running on GCE node
I20260810 18:03:50.562597 10348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:50.563800 10348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:50.564989 10348 hybrid_clock.cc:648] HybridClock initialized: now 1786385030564959 us; error 43 us; skew 500 ppm
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:50.567884 10348 init.cc:377] Logged in from keytab as kudu/127.28.83.124@KRBTEST.COM (short username kudu)
I20260810 18:03:50.568745 10348 webserver.cc:460] Webserver started at http://127.28.83.124:46851/ using document root <none> and password file <none>
I20260810 18:03:50.569329 10348 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:50.569442 10348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:50.569665 10348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:03:50.571692 10348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/data/instance:
uuid: "affd991930dd4571bdf8268399b11254"
format_stamp: "Formatted at 2026-08-10 18:03:50 on dist-test-slave-srgv"
server_key: "d3c18a4da3407cd2d2255b6747697ac3"
server_key_iv: "c3564e41ac3bf1771317e5ce55a18afc"
server_key_version: "encryptionkey@0"
I20260810 18:03:50.572275 10348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/wal/instance:
uuid: "affd991930dd4571bdf8268399b11254"
format_stamp: "Formatted at 2026-08-10 18:03:50 on dist-test-slave-srgv"
server_key: "d3c18a4da3407cd2d2255b6747697ac3"
server_key_iv: "c3564e41ac3bf1771317e5ce55a18afc"
server_key_version: "encryptionkey@0"
I20260810 18:03:50.576076 10348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:03:50.578508 10365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:50.579645 10348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:03:50.579795 10348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/wal
uuid: "affd991930dd4571bdf8268399b11254"
format_stamp: "Formatted at 2026-08-10 18:03:50 on dist-test-slave-srgv"
server_key: "d3c18a4da3407cd2d2255b6747697ac3"
server_key_iv: "c3564e41ac3bf1771317e5ce55a18afc"
server_key_version: "encryptionkey@0"
I20260810 18:03:50.579905 10348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:50.598699 10348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:50.602092 10348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:50.602306 10348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:50.609308 10348 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.124:43279
I20260810 18:03:50.609338 10419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.124:43279 every 8 connection(s)
I20260810 18:03:50.610255 10348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
I20260810 18:03:50.612841 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:50.614271 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 10348
I20260810 18:03:50.614403 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786384863806381-29005-0/minicluster-data/master-2/wal/instance
I20260810 18:03:50.614681 29005 external_mini_cluster.cc:1428] Setting key f9eba067896a56f8f80f714d6d4350e9
I20260810 18:03:50.616997 10420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:50.629057 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.619070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.28.83.126:40389)
0810 18:03:50.619225 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.619230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.619529 (+ 299us) server_negotiation.cc:407] Connection header received
0810 18:03:50.619680 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.619682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.619725 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.619795 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.620675 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.621131 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.622135 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.622392 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.622516 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.625370 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.625376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.625381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.625432 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.627069 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.627606 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.627608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.627610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.627692 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.627985 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.627988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.627989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.628224 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.628330 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.628760 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.628883 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:03:50.629918 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.619006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40389 (local address 127.0.0.1:59622)
0810 18:03:50.619391 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.619441 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:50.619576 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:50.619866 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:50.619871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:50.620287 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:50.620482 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.620497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.621262 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.621266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.622000 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.622013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.622555 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.622560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.622673 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.623196 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:50.623211 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:50.625188 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:50.627183 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.627193 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.627203 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.627483 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.627814 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.627817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.627818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.627892 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.628333 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:50.628338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:50.628513 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 18:03:50.628706 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 18:03:50.629085 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":96,"thread_start_us":107,"threads_started":1}
I20260810 18:03:50.629923 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.616018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.28.83.124:43279)
0810 18:03:50.616438 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.616452 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.616485 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:03:50.617099 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.617115 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.617727 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.618111 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.618959 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.620509 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.621393 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.621920 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.622038 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.624754 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.624762 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.624775 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.624802 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.626851 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.627363 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.627367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.627369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.627468 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.627761 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.627765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.627769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.628314 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.628482 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.628758 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.628986 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
I20260810 18:03:50.631491 10420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:50.640309 10265 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } attempt: 3
I20260810 18:03:50.640887 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.632350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.28.83.125:35271)
0810 18:03:50.632570 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.632575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.632590 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:03:50.632645 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.632649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.632702 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.632784 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.633273 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.634009 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.634931 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.635178 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.635261 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.637532 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.637535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.637537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.637560 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.639040 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.639553 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.639557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.639559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.639650 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.639917 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.639921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.639923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.640178 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.640268 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.640604 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.640734 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260810 18:03:50.640849 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.632224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:35271 (local address 127.0.0.1:60100)
0810 18:03:50.632388 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.632404 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:50.632509 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:50.632824 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:50.632828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:50.633022 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:50.633143 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.633150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.634129 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.634132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.634812 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.634822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.635297 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.635301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.635388 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.635939 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:50.635958 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:50.637319 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:50.639163 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.639167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.639171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.639432 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.639748 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.639750 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.639751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.639822 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.640283 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:50.640287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:50.640390 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 18:03:50.640522 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 18:03:50.640657 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 18:03:50.641995 10420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:03:50.642055 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.632534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.28.83.126:40389)
0810 18:03:50.632658 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.632661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.632870 (+ 209us) server_negotiation.cc:407] Connection header received
0810 18:03:50.632961 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.632963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.632996 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.633056 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.633781 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.634429 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.635331 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.635624 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.635750 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.638234 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.638238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.638240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.638270 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.640063 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.640571 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.640574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.640576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.640674 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.640975 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.640978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.640980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.641193 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.641449 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.641795 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.641911 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:03:50.643903 10341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } attempt: 3
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
I20260810 18:03:50.649567 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.640708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55312)
0810 18:03:50.640993 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.641009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:50.641081 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:50.641380 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:50.641384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:50.641579 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:50.641672 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.641682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.642649 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.642653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.643150 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.643157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.643727 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.643730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.643819 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.644309 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:50.644324 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:50.645851 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:50.647678 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.647682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.647684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.647935 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.648283 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.648289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.648294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.648390 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.648879 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:50.648885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:50.649012 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 18:03:50.649105 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 18:03:50.649253 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
I20260810 18:03:50.650450 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.640970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.28.83.124:43279)
0810 18:03:50.641086 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.641091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.641103 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:03:50.641209 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.641214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.641258 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.641352 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.641800 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.642470 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.643266 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.643595 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.643708 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.645977 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.645980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.645983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.646006 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.647552 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.648071 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.648074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.648076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.648169 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.648520 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.648525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.648529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.648746 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.650091 (+ 1345us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.650184 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.650301 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 18:03:50 dist-test-slave-srgv krb5kdc[10181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385030, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
I20260810 18:03:50.654371 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.642593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55324)
0810 18:03:50.644034 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.644065 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:50.644167 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:50.644433 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:50.644436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:50.644606 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:50.644705 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.644712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.645795 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.645800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.646537 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.646551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.647518 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.647522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.647614 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.648268 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:50.648290 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:50.649678 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:50.652321 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.652323 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.652325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.652529 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.652800 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.652803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.652805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.652877 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.653351 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:50.653353 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:50.653453 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 18:03:50.653811 (+ 358us) client_negotiation.cc:240] Negotiation successful
0810 18:03:50.654204 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829}
I20260810 18:03:50.654480 10265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44: Bootstrap starting.
I20260810 18:03:50.654656 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.643068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.28.83.124:43279)
0810 18:03:50.643519 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.643528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.644165 (+ 637us) server_negotiation.cc:407] Connection header received
0810 18:03:50.644270 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.644276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.644319 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.644417 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.644820 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.645678 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.646671 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.647327 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.647420 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.649798 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.649801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.649804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.649822 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.652216 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.652642 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.652645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.652646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.652714 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.652966 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.652969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.652971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.653211 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.653316 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.654069 (+ 753us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.654476 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":254,"server-negotiator.queue_time_us":378,"thread_start_us":183,"threads_started":1}
I20260810 18:03:50.655728 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.643961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.28.83.125:35271)
0810 18:03:50.644082 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.644123 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.644554 (+ 431us) server_negotiation.cc:407] Connection header received
0810 18:03:50.644686 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.644692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.644746 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.644831 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.645735 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.646188 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.647296 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.647678 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.648106 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.650581 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.650585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.650587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.650604 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.652561 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.653085 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.653088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.653089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.653245 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.653912 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.653918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.653921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.654161 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.655019 (+ 858us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.655244 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.655538 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:03:50.657524 10265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:50.657773 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.645021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43279 (local address 127.0.0.1:55330)
0810 18:03:50.645296 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:50.645312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:50.645389 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:50.647655 (+ 2266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:50.647659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:50.647860 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:50.647981 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.647991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.649211 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.649214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.650040 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:50.650054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.650733 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.650737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:50.650843 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.651268 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:50.651284 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:50.652956 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:50.656039 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.656042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.656044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.656405 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.656722 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:50.656725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:50.656727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:50.656822 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:50.657262 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:50.657267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:50.657366 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 18:03:50.657496 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 18:03:50.657625 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260810 18:03:50.657989 10420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254: Bootstrap starting.
I20260810 18:03:50.658468 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.645568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.28.83.124:43279)
0810 18:03:50.647418 (+ 1850us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.647426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.647437 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:03:50.647468 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.647470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.647508 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.648109 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.648249 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.649075 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.650191 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.650580 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.651038 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.653103 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.653107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.653109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.653135 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.655739 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.656528 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.656531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.656532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.656612 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.656935 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.656940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.656943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.657130 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.658208 (+ 1078us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.658271 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.658355 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1780,"thread_start_us":106,"threads_started":1}
I20260810 18:03:50.658433 10265 log.cc:824] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:50.660251 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:03:50.660282 10420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:50.660642 10265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44: No bootstrap required, opened a new log
I20260810 18:03:50.660856 10420 log.cc:824] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:50.663057 10420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254: No bootstrap required, opened a new log
I20260810 18:03:50.663960 10341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b: Bootstrap starting.
I20260810 18:03:50.664918 10265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } }
I20260810 18:03:50.665132 10265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:50.665195 10265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800bbce541c34bd38b56ce4043d5df44, State: Initialized, Role: FOLLOWER
I20260810 18:03:50.665669 10265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } }
I20260810 18:03:50.666131 10436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.666317 10436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:03:50.666532 10341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:50.666941 10265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:03:50.667454 10341 log.cc:824] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:50.667984 10420 raft_consensus.cc:348] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } }
I20260810 18:03:50.668246 10420 raft_consensus.cc:374] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:50.668319 10420 raft_consensus.cc:729] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affd991930dd4571bdf8268399b11254, State: Initialized, Role: FOLLOWER
I20260810 18:03:50.668769 10420 consensus_queue.cc:260] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } }
W20260810 18:03:50.668959 10215 tablet.cc:2369] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:03:50.669888 10341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b: No bootstrap required, opened a new log
I20260810 18:03:50.670400 10420 sys_catalog.cc:564] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:03:50.673959 10341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } }
I20260810 18:03:50.674263 10341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:50.674394 10341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901f8a5b2bb949f7af54b1ee547c1b5b, State: Initialized, Role: FOLLOWER
I20260810 18:03:50.672042 10194 master_runner.cc:186] Error getting master registration for 127.28.83.125:35271: OK, instance_id { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" instance_seqno: 1786385030386403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:03:50.676555 10194 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 18:03:50.679207 10448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:03:50.679277 10448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 18:03:50.679257 10461 catalog_manager.cc:1558] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:03:50.679740 10461 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:03:50.679639 10449 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.679840 10449 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:03:50.678355 10348 master_runner.cc:186] Error getting master registration for 127.28.83.125:35271: OK, instance_id { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" instance_seqno: 1786385030386403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:03:50.681500 10348 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:03:50.682530 10341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } }
I20260810 18:03:50.684105 10341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:03:50.684608 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.662680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.28.83.126:40389)
0810 18:03:50.662787 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.662791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.668527 (+ 5736us) server_negotiation.cc:407] Connection header received
0810 18:03:50.669637 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.669640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.669685 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.669734 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.672543 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.673288 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.676469 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.676812 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.676885 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.680225 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.680231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.680236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.680263 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.682250 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.682744 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.682749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.682754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.682841 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.683231 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.683235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.683237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.683523 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.683654 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.684306 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.684424 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:03:50.685159 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.663924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.28.83.124:43279)
0810 18:03:50.664026 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.664031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.668970 (+ 4939us) server_negotiation.cc:407] Connection header received
0810 18:03:50.669123 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.669126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.669172 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.669249 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.673297 (+ 4048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.674150 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.675552 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.675884 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.675961 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.681198 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.681201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.681204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.681230 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.682969 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.683474 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.683478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.683479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.683566 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.683890 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.683894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.683896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.684135 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.684282 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.684873 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.684998 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:03:50.685436 10464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.685601 10464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:03:50.686906 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:50.663361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.28.83.125:35271)
0810 18:03:50.663459 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:50.663463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:50.668759 (+ 5296us) server_negotiation.cc:407] Connection header received
0810 18:03:50.669354 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:50.669360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:50.669434 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:50.669489 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:50.671648 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.672403 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.678433 (+ 6030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:50.678785 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:50.678864 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:50.681221 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:50.681225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:50.681228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:50.681249 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:50.683173 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.683812 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.683815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.683817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.683899 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:50.684247 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:50.684250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:50.684252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:50.684461 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:50.684574 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:50.686608 (+ 2034us) server_negotiation.cc:299] Negotiation successful
0810 18:03:50.686764 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 18:03:50.689487 10475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:03:50.689563 10475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:03:50.693219 10272 master_runner.cc:263] No leader master found from master 901f8a5b2bb949f7af54b1ee547c1b5b
I20260810 18:03:50.693296 10272 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:03:50.714838 10436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:03:50.715046 10436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } }
I20260810 18:03:50.716151 10436 leader_election.cc:288] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 901f8a5b2bb949f7af54b1ee547c1b5b (127.28.83.125:35271), affd991930dd4571bdf8268399b11254 (127.28.83.124:43279)
I20260810 18:03:50.716887 10313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800bbce541c34bd38b56ce4043d5df44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" is_pre_election: true
I20260810 18:03:50.717218 10313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800bbce541c34bd38b56ce4043d5df44 in term 0.
I20260810 18:03:50.717662 10213 leader_election.cc:302] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 800bbce541c34bd38b56ce4043d5df44, 901f8a5b2bb949f7af54b1ee547c1b5b; no voters:
I20260810 18:03:50.718318 10392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800bbce541c34bd38b56ce4043d5df44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affd991930dd4571bdf8268399b11254" is_pre_election: true
I20260810 18:03:50.718613 10392 raft_consensus.cc:2454] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800bbce541c34bd38b56ce4043d5df44 in term 0.
I20260810 18:03:50.719188 10436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:03:50.719285 10436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:03:50.719337 10436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:50.720731 10436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } }
I20260810 18:03:50.721063 10436 leader_election.cc:288] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [CANDIDATE]: Term 1 election: Requested vote from peers 901f8a5b2bb949f7af54b1ee547c1b5b (127.28.83.125:35271), affd991930dd4571bdf8268399b11254 (127.28.83.124:43279)
I20260810 18:03:50.721511 10313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800bbce541c34bd38b56ce4043d5df44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b"
I20260810 18:03:50.721637 10313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:50.722918 10313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800bbce541c34bd38b56ce4043d5df44 in term 1.
I20260810 18:03:50.723229 10213 leader_election.cc:302] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 800bbce541c34bd38b56ce4043d5df44, 901f8a5b2bb949f7af54b1ee547c1b5b; no voters:
I20260810 18:03:50.723587 10392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800bbce541c34bd38b56ce4043d5df44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affd991930dd4571bdf8268399b11254"
I20260810 18:03:50.723712 10392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:50.724963 10392 raft_consensus.cc:2454] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800bbce541c34bd38b56ce4043d5df44 in term 1.
I20260810 18:03:50.727682 10436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:03:50.727946 10436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [term 1 LEADER]: Becoming Leader. State: Replica: 800bbce541c34bd38b56ce4043d5df44, State: Running, Role: LEADER
I20260810 18:03:50.728355 10436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } }
I20260810 18:03:50.730379 10436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800bbce541c34bd38b56ce4043d5df44. Latest consensus state: current_term: 1 leader_uuid: "800bbce541c34bd38b56ce4043d5df44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.730494 10436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:50.732420 10479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:03:50.735808 10479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:03:50.742430 10392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [term 1 FOLLOWER]: Refusing update from remote peer 800bbce541c34bd38b56ce4043d5df44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:03:50.743125 10477 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:03:50.747344 10449 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800bbce541c34bd38b56ce4043d5df44. Latest consensus state: current_term: 1 leader_uuid: "800bbce541c34bd38b56ce4043d5df44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.747466 10449 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:03:50.750622 10313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [term 1 FOLLOWER]: Refusing update from remote peer 800bbce541c34bd38b56ce4043d5df44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:03:50.751230 10477 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:03:50.754784 10464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800bbce541c34bd38b56ce4043d5df44. Latest consensus state: current_term: 1 leader_uuid: "800bbce541c34bd38b56ce4043d5df44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.754904 10464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:03:50.761443 10477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "800bbce541c34bd38b56ce4043d5df44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.761581 10477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:50.762801 10464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "800bbce541c34bd38b56ce4043d5df44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.762930 10464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:03:50.763324 10480 mvcc.cc:205] Tried to move back new op lower bound from 7317033085912866816 to 7317033085867503616. Current Snapshot: MvccSnapshot[applied={T|T < 7317033085912866816}]
I20260810 18:03:50.765033 10483 mvcc.cc:205] Tried to move back new op lower bound from 7317033085912866816 to 7317033085867503616. Current Snapshot: MvccSnapshot[applied={T|T < 7317033085912866816}]
I20260810 18:03:50.767565 10436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "800bbce541c34bd38b56ce4043d5df44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.767691 10436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44 [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:50.768801 10449 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "800bbce541c34bd38b56ce4043d5df44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "800bbce541c34bd38b56ce4043d5df44" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40389 } } peers { permanent_uuid: "901f8a5b2bb949f7af54b1ee547c1b5b" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 35271 } } peers { permanent_uuid: "affd991930dd4571bdf8268399b11254" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43279 } } }
I20260810 18:03:50.768929 10449 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:03:50.770867 10481 mvcc.cc:205] Tried to move back new op lower bound from 7317033085912866816 to 7317033085867503616. Current Snapshot: MvccSnapshot[applied={T|T < 7317033085912866816}]
I20260810 18:03:50.776203 10479 catalog_manager.cc:1347] Generated new cluster ID: 2e8cea7f9c90496ba532cb59949539ce
I20260810 18:03:50.776273 10479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:03:50.817168 10479 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:03:50.818536 10479 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:03:50.838549 10479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 800bbce541c34bd38b56ce4043d5df44: Generated new TSK 0
I20260810 18:03:50.839432 10479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:03:51.681007 10461 catalog_manager.cc:1259] Loaded cluster ID: 2e8cea7f9c90496ba532cb59949539ce
I20260810 18:03:51.681097 10461 catalog_manager.cc:1552] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254: loading cluster ID for follower catalog manager: success
I20260810 18:03:51.682745 10461 catalog_manager.cc:1574] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254: acquiring CA information for follower catalog manager: success
I20260810 18:03:51.683985 10461 catalog_manager.cc:1602] T 00000000000000000000000000000000 P affd991930dd4571bdf8268399b11254: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:03:51.690677 10475 catalog_manager.cc:1259] Loaded cluster ID: 2e8cea7f9c90496ba532cb59949539ce
I20260810 18:03:51.690761 10475 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b: loading cluster ID for follower catalog manager: success
I20260810 18:03:51.692267 10475 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b: acquiring CA information for follower catalog manager: success
I20260810 18:03:51.693308 10475 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 901f8a5b2bb949f7af54b1ee547c1b5b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:03:51.916374 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 10194
I20260810 18:03:51.923846 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 10272
I20260810 18:03:51.930388 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 10348
2026-08-10T18:03:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3120 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:03:51 dist-test-slave-srgv krb5kdc[10495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:03:51 dist-test-slave-srgv krb5kdc[10495](info): set up 2 sockets
Aug 10 18:03:51 dist-test-slave-srgv krb5kdc[10495](info): commencing operation
krb5kdc: starting...
W20260810 18:03:52.979990 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:03:53 dist-test-slave-srgv krb5kdc[10495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:03:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:03:53.104790 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33161
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:33161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:53.214155 10509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:53.214454 10509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:53.214532 10509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:53.218595 10509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:03:53.218680 10509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:53.218717 10509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:53.218772 10509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:03:53.218811 10509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:03:53.224128 10509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:33161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:53.225199 10509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:53.226274 10509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:53.235145 10514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:53.235137 10515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:53.235263 10517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:53.235472 10509 server_base.cc:1034] running on GCE node
I20260810 18:03:53.236057 10509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:53.237112 10509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:53.238313 10509 hybrid_clock.cc:648] HybridClock initialized: now 1786385033238277 us; error 56 us; skew 500 ppm
Aug 10 18:03:53 dist-test-slave-srgv krb5kdc[10495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:53.241117 10509 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:03:53.242049 10509 webserver.cc:460] Webserver started at http://127.28.83.126:42927/ using document root <none> and password file <none>
I20260810 18:03:53.242614 10509 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:53.242708 10509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:53.242934 10509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:03:53.245003 10509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "e75d910e950a40389618299de2ea2bec"
format_stamp: "Formatted at 2026-08-10 18:03:53 on dist-test-slave-srgv"
server_key: "8b381448d5cee72883d3f3d879f84616"
server_key_iv: "a8d48d58a11f68d33b677f08e355fe41"
server_key_version: "encryptionkey@0"
I20260810 18:03:53.245679 10509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "e75d910e950a40389618299de2ea2bec"
format_stamp: "Formatted at 2026-08-10 18:03:53 on dist-test-slave-srgv"
server_key: "8b381448d5cee72883d3f3d879f84616"
server_key_iv: "a8d48d58a11f68d33b677f08e355fe41"
server_key_version: "encryptionkey@0"
I20260810 18:03:53.249346 10509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:03:53.251664 10524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:53.252877 10509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:03:53.253018 10509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "e75d910e950a40389618299de2ea2bec"
format_stamp: "Formatted at 2026-08-10 18:03:53 on dist-test-slave-srgv"
server_key: "8b381448d5cee72883d3f3d879f84616"
server_key_iv: "a8d48d58a11f68d33b677f08e355fe41"
server_key_version: "encryptionkey@0"
I20260810 18:03:53.253134 10509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:53.266685 10509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:53.269614 10509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:53.269814 10509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:53.276859 10509 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:33161
I20260810 18:03:53.276888 10577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:33161 every 8 connection(s)
I20260810 18:03:53.277874 10509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:03:53.280617 10578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:53.281054 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 10509
I20260810 18:03:53.281178 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:03:53.281490 29005 external_mini_cluster.cc:1428] Setting key a1123e62ffe4cd02a9f9d9f253d26c3c
I20260810 18:03:53.287665 10578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec: Bootstrap starting.
I20260810 18:03:53.290362 10578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec: Neither blocks nor log segments found. Creating new log.
Aug 10 18:03:53 dist-test-slave-srgv krb5kdc[10495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:53.291131 10578 log.cc:824] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:53.292941 10578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec: No bootstrap required, opened a new log
I20260810 18:03:53.296041 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:53.282846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.28.83.126:33161)
0810 18:03:53.283269 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:53.283283 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:53.283318 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:03:53.284141 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:53.284159 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:53.284473 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:53.284810 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:53.285756 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:53.286710 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:53.287876 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:53.288326 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:53.288446 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:53.291373 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:53.291379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:53.291391 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:53.291418 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:53.293220 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:53.293714 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:53.293720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:53.293727 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:53.293841 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:53.294183 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:53.294186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:53.294188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:53.294565 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:53.294776 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:53.295075 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 18:03:53.295360 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
I20260810 18:03:53.296556 10578 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d910e950a40389618299de2ea2bec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33161 } }
I20260810 18:03:53.296797 10578 raft_consensus.cc:374] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:53.296877 10578 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75d910e950a40389618299de2ea2bec, State: Initialized, Role: FOLLOWER
I20260810 18:03:53.297358 10578 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d910e950a40389618299de2ea2bec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33161 } }
I20260810 18:03:53.297534 10578 raft_consensus.cc:388] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:03:53.297601 10578 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:03:53.297693 10578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:53.298810 10578 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d910e950a40389618299de2ea2bec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33161 } }
I20260810 18:03:53.299013 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:03:53.299186 10578 leader_election.cc:302] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e75d910e950a40389618299de2ea2bec; no voters:
I20260810 18:03:53.299533 10578 leader_election.cc:288] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:03:53.299662 10583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:03:53.299922 10583 raft_consensus.cc:686] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [term 1 LEADER]: Becoming Leader. State: Replica: e75d910e950a40389618299de2ea2bec, State: Running, Role: LEADER
I20260810 18:03:53.300276 10583 consensus_queue.cc:237] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d910e950a40389618299de2ea2bec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33161 } }
I20260810 18:03:53.300848 10578 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:03:53.301525 10583 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75d910e950a40389618299de2ea2bec. Latest consensus state: current_term: 1 leader_uuid: "e75d910e950a40389618299de2ea2bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d910e950a40389618299de2ea2bec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33161 } } }
I20260810 18:03:53.301745 10583 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:53.301956 10585 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e75d910e950a40389618299de2ea2bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d910e950a40389618299de2ea2bec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33161 } } }
I20260810 18:03:53.302114 10585 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:53.302201 10593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:03:53.305346 10593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:03:53.310847 10593 catalog_manager.cc:1347] Generated new cluster ID: ecc6340c5a1b458a88ad7e99483add9a
I20260810 18:03:53.310913 10593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:03:53.325466 10593 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:03:53.326375 10593 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:03:53.330945 10593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e75d910e950a40389618299de2ea2bec: Generated new TSK 0
I20260810 18:03:53.331570 10593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:03:53.355719 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 10509
2026-08-10T18:03:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1422 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:03:53 dist-test-slave-srgv krb5kdc[10606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:03:53 dist-test-slave-srgv krb5kdc[10606](info): set up 2 sockets
Aug 10 18:03:53 dist-test-slave-srgv krb5kdc[10606](info): commencing operation
krb5kdc: starting...
W20260810 18:03:54.412400 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:03:54 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:03:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:03:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:03:54.534209 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36225
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:54.646008 10619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:54.646298 10619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:54.646340 10619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:54.650429 10619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:03:54.650489 10619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:54.650508 10619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:54.650578 10619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:03:54.650604 10619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:03:54.655970 10619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:54.657070 10619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:54.658159 10619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:54.666694 10625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:54.666754 10624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:54.666730 10619 server_base.cc:1034] running on GCE node
W20260810 18:03:54.666693 10627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:54.667512 10619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:54.668602 10619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:54.669802 10619 hybrid_clock.cc:648] HybridClock initialized: now 1786385034669774 us; error 43 us; skew 500 ppm
Aug 10 18:03:54 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385034, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:54.676046 10619 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:03:54.677017 10619 webserver.cc:460] Webserver started at http://127.28.83.126:34403/ using document root <none> and password file <none>
I20260810 18:03:54.677650 10619 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:54.677745 10619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:54.677979 10619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:03:54.680009 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "78d6054ec43d469d872a7c92b733fb97"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "d1164d3346585ef62449d0787544fc6e"
server_key_iv: "744eed96e0aa769e3946dc03263ba4ce"
server_key_version: "encryptionkey@0"
I20260810 18:03:54.680593 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "78d6054ec43d469d872a7c92b733fb97"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "d1164d3346585ef62449d0787544fc6e"
server_key_iv: "744eed96e0aa769e3946dc03263ba4ce"
server_key_version: "encryptionkey@0"
I20260810 18:03:54.684327 10619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:03:54.686765 10634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:54.687907 10619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:03:54.688050 10619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "78d6054ec43d469d872a7c92b733fb97"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "d1164d3346585ef62449d0787544fc6e"
server_key_iv: "744eed96e0aa769e3946dc03263ba4ce"
server_key_version: "encryptionkey@0"
I20260810 18:03:54.688159 10619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:54.709450 10619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:54.712605 10619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:54.712829 10619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:54.720224 10619 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:36225
I20260810 18:03:54.720536 10688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:36225 every 8 connection(s)
I20260810 18:03:54.721246 10619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:03:54.723944 10689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:54.729241 10689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Bootstrap starting.
I20260810 18:03:54.731114 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 10619
I20260810 18:03:54.731256 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:03:54.731550 29005 external_mini_cluster.cc:1428] Setting key fb3c67196c7274dc0e63fa525f6ed644
I20260810 18:03:54.731953 10689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:54.732785 10689 log.cc:824] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:54.734725 10689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: No bootstrap required, opened a new log
I20260810 18:03:54.738744 10689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:54.739027 10689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:54.739080 10689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d6054ec43d469d872a7c92b733fb97, State: Initialized, Role: FOLLOWER
I20260810 18:03:54.739609 10689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:54.739776 10689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:03:54.739835 10689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:03:54.739946 10689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 18:03:54 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:54.740887 10689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:54.741262 10689 leader_election.cc:302] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d6054ec43d469d872a7c92b733fb97; no voters:
I20260810 18:03:54.741684 10689 leader_election.cc:288] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:03:54.741845 10694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:03:54.742141 10694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 1 LEADER]: Becoming Leader. State: Replica: 78d6054ec43d469d872a7c92b733fb97, State: Running, Role: LEADER
I20260810 18:03:54.742514 10694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:54.743058 10689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:03:54.743691 10694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d6054ec43d469d872a7c92b733fb97. Latest consensus state: current_term: 1 leader_uuid: "78d6054ec43d469d872a7c92b733fb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } } }
I20260810 18:03:54.743891 10694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:54.744175 10696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d6054ec43d469d872a7c92b733fb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } } }
I20260810 18:03:54.744295 10696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:54.744386 10702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:03:54.747251 10702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:03:54.747411 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:54.732785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.28.83.126:36225)
0810 18:03:54.733260 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:54.733277 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:54.733311 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:03:54.733995 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:54.734012 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:54.734322 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:54.734590 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:54.735493 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:54.736471 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:54.737750 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:54.738198 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:54.738327 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:54.741193 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:54.741199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:54.741215 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:54.741245 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:54.743469 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:54.744067 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:54.744076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:54.744085 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:54.744240 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:54.744581 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:54.744584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:54.744586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:54.745158 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:54.746046 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:54.746193 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 18:03:54.746497 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":170,"threads_started":1}
I20260810 18:03:54.753268 10702 catalog_manager.cc:1347] Generated new cluster ID: 3a16610d2fca4f50afab55c7e78c32f7
I20260810 18:03:54.753335 10702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:03:54.764940 10702 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:03:54.766129 10702 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:03:54.775071 10702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Generated new TSK 0
I20260810 18:03:54.775758 10702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:03:54.810457 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36225
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:54.925254 10715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:54.925633 10715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:54.925727 10715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:54.933023 10715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:54.933130 10715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:54.933245 10715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:03:54.940922 10715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:54.942444 10715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:54.943886 10715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:54.956903 10721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:54.957046 10715 server_base.cc:1034] running on GCE node
W20260810 18:03:54.956902 10720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:54.957147 10723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:54.957685 10715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:54.958609 10715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:54.959827 10715 hybrid_clock.cc:648] HybridClock initialized: now 1786385034959808 us; error 47 us; skew 500 ppm
Aug 10 18:03:54 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385034, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:54.963770 10715 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:03:54.964932 10715 webserver.cc:460] Webserver started at http://127.28.83.65:43239/ using document root <none> and password file <none>
I20260810 18:03:54.965677 10715 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:54.965795 10715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:54.966037 10715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:03:54.968791 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "ffe36616bec44765a949d3a96e04952a"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "b82ffd703d4b6461d7512a57b7a9453d"
server_key_iv: "7accbaba1fc73c905b41264192056ece"
server_key_version: "encryptionkey@0"
I20260810 18:03:54.969502 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "ffe36616bec44765a949d3a96e04952a"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "b82ffd703d4b6461d7512a57b7a9453d"
server_key_iv: "7accbaba1fc73c905b41264192056ece"
server_key_version: "encryptionkey@0"
I20260810 18:03:54.974205 10715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:03:54.976960 10731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:54.978513 10715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:03:54.978631 10715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "ffe36616bec44765a949d3a96e04952a"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "b82ffd703d4b6461d7512a57b7a9453d"
server_key_iv: "7accbaba1fc73c905b41264192056ece"
server_key_version: "encryptionkey@0"
I20260810 18:03:54.978753 10715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:54.995620 10715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:54.999042 10715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:54.999199 10715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:54.999742 10715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:55.000515 10715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:55.000560 10715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.000602 10715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:55.000623 10715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.009614 10715 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:36635
I20260810 18:03:55.009651 10849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:36635 every 8 connection(s)
I20260810 18:03:55.010505 10715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:03:55.017679 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 10715
I20260810 18:03:55.017831 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:03:55.018189 29005 external_mini_cluster.cc:1428] Setting key 9205d75a17614e4bfd7b007d9d836f17
Aug 10 18:03:55 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385034, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:55.024755 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.012230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:39801 (local address 127.28.83.126:36225)
0810 18:03:55.012413 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.012418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.013217 (+ 799us) server_negotiation.cc:407] Connection header received
0810 18:03:55.014549 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.014552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.014598 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.014674 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:55.015769 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.016292 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.017473 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.017721 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.017841 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.020584 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:55.020591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:55.020597 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:55.020630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:55.022311 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:55.022971 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:55.022977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:55.022981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:55.023075 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:55.023408 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:55.023414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:55.023418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:55.023680 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:55.023796 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.024285 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.024439 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 18:03:55.025449 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.012600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:39801)
0810 18:03:55.013049 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:55.013092 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:55.014360 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:55.014840 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:55.014846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:55.015193 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:55.015639 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.015652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.016397 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.016399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.017314 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.017323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.017931 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.017939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.018107 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.018773 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:55.018797 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:55.020413 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:55.022433 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:55.022441 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:55.022458 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:55.022866 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:55.023168 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:55.023170 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:55.023171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:55.023310 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:55.023797 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:55.023802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:55.024024 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 18:03:55.024191 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 18:03:55.024441 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260810 18:03:55.026780 10850 heartbeater.cc:347] Connected to a master server at 127.28.83.126:36225
I20260810 18:03:55.027040 10850 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:55.027521 10850 heartbeater.cc:511] Master 127.28.83.126:36225 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:03:55.029017 10653 ts_manager.cc:194] Registered new tserver with Master: ffe36616bec44765a949d3a96e04952a (127.28.83.65:36635)
I20260810 18:03:55.030333 10653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:39801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:03:55.048483 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36225
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:55.171384 10855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:55.171787 10855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:55.171873 10855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:55.178773 10855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:55.178843 10855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:55.178947 10855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:03:55.184547 10855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:55.185756 10855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:55.186882 10855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:55.196521 10863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:55.196533 10860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:55.196533 10861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:55.196875 10855 server_base.cc:1034] running on GCE node
I20260810 18:03:55.197309 10855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:55.198015 10855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:55.199277 10855 hybrid_clock.cc:648] HybridClock initialized: now 1786385035199258 us; error 30 us; skew 500 ppm
Aug 10 18:03:55 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:55.202186 10855 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:03:55.203085 10855 webserver.cc:460] Webserver started at http://127.28.83.66:36897/ using document root <none> and password file <none>
I20260810 18:03:55.203665 10855 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:55.203734 10855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:55.203982 10855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:03:55.206184 10855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "fe73cd93bb5d4cae839fb9f590709106"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "6d631932a5587d5bb8d8f0dbd16d736a"
server_key_iv: "1fb3f6cddd35fbd51042d6bafd139b89"
server_key_version: "encryptionkey@0"
I20260810 18:03:55.206785 10855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "fe73cd93bb5d4cae839fb9f590709106"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "6d631932a5587d5bb8d8f0dbd16d736a"
server_key_iv: "1fb3f6cddd35fbd51042d6bafd139b89"
server_key_version: "encryptionkey@0"
I20260810 18:03:55.210626 10855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:03:55.213104 10872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.214365 10855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:03:55.214489 10855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "fe73cd93bb5d4cae839fb9f590709106"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "6d631932a5587d5bb8d8f0dbd16d736a"
server_key_iv: "1fb3f6cddd35fbd51042d6bafd139b89"
server_key_version: "encryptionkey@0"
I20260810 18:03:55.214596 10855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:55.232276 10855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:55.235347 10855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:55.235574 10855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:55.236194 10855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:55.237052 10855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:55.237100 10855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.237174 10855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:55.237218 10855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.247216 10855 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:41333
I20260810 18:03:55.247282 10990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:41333 every 8 connection(s)
I20260810 18:03:55.248126 10855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:03:55.249162 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 10855
I20260810 18:03:55.249296 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:03:55.249610 29005 external_mini_cluster.cc:1428] Setting key 474933188f72577192f2daf1fb475940
Aug 10 18:03:55 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:03:55.262986 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.250093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:59895 (local address 127.28.83.126:36225)
0810 18:03:55.250256 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.250261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.251163 (+ 902us) server_negotiation.cc:407] Connection header received
0810 18:03:55.252507 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.252513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.252576 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.252674 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:55.254274 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.255091 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.256320 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.256739 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.256865 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.259381 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:55.259385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:55.259390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:55.259421 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:55.260895 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:55.261473 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:55.261481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:55.261482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:55.261556 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:55.261859 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:55.261861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:55.261862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:55.262027 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:55.262144 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.262709 (+ 565us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.262829 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 18:03:55.264185 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.250450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:59895)
0810 18:03:55.250994 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:55.251055 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:55.252297 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:55.252821 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:55.252828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:55.253374 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:55.254102 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.254124 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.255228 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.255231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.256186 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.256196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.256903 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.256905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.257042 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.257734 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:55.257750 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:55.259205 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:55.261027 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:55.261032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:55.261043 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:55.261322 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:55.261641 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:55.261644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:55.261645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:55.261777 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:55.262144 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:55.262149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:55.262420 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 18:03:55.262652 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 18:03:55.263034 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20260810 18:03:55.265429 10991 heartbeater.cc:347] Connected to a master server at 127.28.83.126:36225
I20260810 18:03:55.265684 10991 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:55.266160 10991 heartbeater.cc:511] Master 127.28.83.126:36225 requested a full tablet report, sending...
I20260810 18:03:55.267277 10653 ts_manager.cc:194] Registered new tserver with Master: fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333)
I20260810 18:03:55.267835 10653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:59895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:03:55.279783 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36225
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:55.393054 10996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:55.393316 10996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:55.393394 10996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:55.397490 10996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:55.397574 10996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:55.397634 10996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:03:55.402748 10996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:55.403746 10996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:55.404721 10996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:55.414674 11003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:55.414745 10996 server_base.cc:1034] running on GCE node
W20260810 18:03:55.414702 11001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:55.414691 11005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:55.415388 10996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:55.415980 10996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:55.417178 10996 hybrid_clock.cc:648] HybridClock initialized: now 1786385035417158 us; error 31 us; skew 500 ppm
Aug 10 18:03:55 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:55.420186 10996 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:03:55.421056 10996 webserver.cc:460] Webserver started at http://127.28.83.67:36021/ using document root <none> and password file <none>
I20260810 18:03:55.421705 10996 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:55.421803 10996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:55.422031 10996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:03:55.424118 10996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "8451c71ffbea48ce91d5efd293c37643"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "30ad38ededfc08bc69be84b6cffe2a42"
server_key_iv: "c9c16924f16c364e38b8373f05e47e01"
server_key_version: "encryptionkey@0"
I20260810 18:03:55.424769 10996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "8451c71ffbea48ce91d5efd293c37643"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "30ad38ededfc08bc69be84b6cffe2a42"
server_key_iv: "c9c16924f16c364e38b8373f05e47e01"
server_key_version: "encryptionkey@0"
I20260810 18:03:55.428434 10996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:03:55.430801 11012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.432021 10996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:03:55.432185 10996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "8451c71ffbea48ce91d5efd293c37643"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "30ad38ededfc08bc69be84b6cffe2a42"
server_key_iv: "c9c16924f16c364e38b8373f05e47e01"
server_key_version: "encryptionkey@0"
I20260810 18:03:55.432300 10996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:55.449280 10996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:55.452211 10996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:55.452425 10996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:55.453020 10996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:55.453877 10996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:55.453953 10996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.454022 10996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:55.454074 10996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.463485 10996 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:46625
I20260810 18:03:55.463511 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:46625 every 8 connection(s)
I20260810 18:03:55.464433 10996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:03:55.466347 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 10996
I20260810 18:03:55.466475 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:03:55.466763 29005 external_mini_cluster.cc:1428] Setting key 1a8712c7c7d622964394ae9ce5d40068
Aug 10 18:03:55 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:55.478174 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.466419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:45073 (local address 127.28.83.126:36225)
0810 18:03:55.467163 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.467168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.467530 (+ 362us) server_negotiation.cc:407] Connection header received
0810 18:03:55.468984 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.468987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.469039 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.469139 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:55.470383 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.470921 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.472053 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.472279 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.472394 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.474937 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:55.474940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:55.474942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:55.474964 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:55.476211 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:55.476701 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:55.476705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:55.476706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:55.476790 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:55.477120 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:55.477123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:55.477125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:55.477298 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:55.477408 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.477902 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.478017 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627}
I20260810 18:03:55.478991 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.466867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:45073)
0810 18:03:55.467388 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:55.467432 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:55.468779 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:55.469315 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:55.469323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:55.469749 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:55.470241 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.470255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.471052 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.471056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.471926 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.471935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.472435 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.472439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.472607 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.473227 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:55.473244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:55.474762 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:55.476336 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:55.476341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:55.476354 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:55.476597 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:55.476883 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:55.476887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:55.476889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:55.477023 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:55.477434 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:55.477443 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:55.477676 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 18:03:55.477863 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 18:03:55.478188 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":149,"threads_started":1}
I20260810 18:03:55.480254 11132 heartbeater.cc:347] Connected to a master server at 127.28.83.126:36225
I20260810 18:03:55.480505 11132 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:55.481024 11132 heartbeater.cc:511] Master 127.28.83.126:36225 requested a full tablet report, sending...
I20260810 18:03:55.482060 10652 ts_manager.cc:194] Registered new tserver with Master: 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625)
I20260810 18:03:55.482714 10652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:45073
I20260810 18:03:55.492479 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:03:55.501955 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.494493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.28.83.126:36225)
0810 18:03:55.494616 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.494620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.494781 (+ 161us) server_negotiation.cc:407] Connection header received
0810 18:03:55.494879 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.494881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.494918 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.494997 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:55.495709 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.496206 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.496986 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.497183 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.497262 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.498656 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:55.498659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:55.498661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:55.498679 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:55.500179 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:55.500557 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:55.500560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:55.500561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:55.500635 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:55.500860 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:55.500862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:55.500863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:55.501282 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:55.501469 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.501641 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.501747 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:03:55.504906 10652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:03:55.508009 10652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:03:55.522750 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.517285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:36635 (local address 127.0.0.1:56176)
0810 18:03:55.517645 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:55.517662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:55.517749 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:55.518516 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:55.518522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:55.518543 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:55.518631 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.518637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.519932 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.519935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.521257 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.521286 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.522382 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.522388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.522484 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.522503 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:03:55.522570 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 18:03:55.522619 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
I20260810 18:03:55.522818 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.517748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56176 (local address 127.28.83.65:36635)
0810 18:03:55.518056 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.518073 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.518096 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:03:55.518165 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.518170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.518378 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.518524 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:55.518728 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.519796 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.521443 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.522260 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.522357 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.522483 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.522594 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.522675 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":106,"threads_started":1}
I20260810 18:03:55.522753 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.516878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.28.83.66:41333)
0810 18:03:55.517297 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.517310 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.517661 (+ 351us) server_negotiation.cc:407] Connection header received
0810 18:03:55.517945 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.517951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.518169 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.518336 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:55.518630 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.519704 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.521146 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.522010 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.522222 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.522371 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.522482 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.522590 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":104,"threads_started":1}
I20260810 18:03:55.522751 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.516437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41333 (local address 127.0.0.1:38610)
0810 18:03:55.517551 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:55.517566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:55.517731 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:55.518328 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:55.518330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:55.518348 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:55.518452 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.518460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.519867 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.519870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.521014 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.521024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.522157 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.522161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.522290 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.522330 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 18:03:55.522451 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 18:03:55.522582 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":420,"thread_start_us":105,"threads_started":1}
I20260810 18:03:55.523612 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.516427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.28.83.67:46625)
0810 18:03:55.517356 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.517367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.517388 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:03:55.517523 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.517530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.517747 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.517912 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:55.518448 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.519671 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.521463 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.522006 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.522152 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.522306 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.523343 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.523453 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":183,"threads_started":1}
I20260810 18:03:55.523183 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.516279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:46625 (local address 127.0.0.1:54970)
0810 18:03:55.516808 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:55.516862 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:55.516992 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:55.518140 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:55.518146 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:55.518169 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:55.518301 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.518307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.519829 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.519833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.521280 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.521286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.522150 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.522154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.522903 (+ 749us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.522936 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 18:03:55.523012 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:03:55.523068 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":181,"thread_start_us":93,"threads_started":1}
I20260810 18:03:55.526120 10921 tablet_service.cc:1467] Processing CreateTablet for tablet 31518501c1524a19a03bf18aa0c99d9f (DEFAULT_TABLE table=test-table [id=c2322eea7647445ab69060f1f66f8754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:03:55.526214 10781 tablet_service.cc:1467] Processing CreateTablet for tablet 31518501c1524a19a03bf18aa0c99d9f (DEFAULT_TABLE table=test-table [id=c2322eea7647445ab69060f1f66f8754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:03:55.526496 11063 tablet_service.cc:1467] Processing CreateTablet for tablet 31518501c1524a19a03bf18aa0c99d9f (DEFAULT_TABLE table=test-table [id=c2322eea7647445ab69060f1f66f8754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:03:55.527336 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31518501c1524a19a03bf18aa0c99d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:55.527334 10921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31518501c1524a19a03bf18aa0c99d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:55.527348 11063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31518501c1524a19a03bf18aa0c99d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:55.533567 11149 tablet_bootstrap.cc:492] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: Bootstrap starting.
I20260810 18:03:55.534263 11151 tablet_bootstrap.cc:492] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: Bootstrap starting.
I20260810 18:03:55.536654 11149 tablet_bootstrap.cc:654] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:55.536748 11151 tablet_bootstrap.cc:654] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:55.537676 11151 log.cc:824] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:55.537744 11150 tablet_bootstrap.cc:492] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Bootstrap starting.
I20260810 18:03:55.538044 11149 log.cc:824] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:55.539531 11150 tablet_bootstrap.cc:654] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:55.539894 11149 tablet_bootstrap.cc:492] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: No bootstrap required, opened a new log
I20260810 18:03:55.540092 11149 ts_tablet_manager.cc:1397] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260810 18:03:55.540318 11150 log.cc:824] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:55.542184 11150 tablet_bootstrap.cc:492] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: No bootstrap required, opened a new log
I20260810 18:03:55.542393 11150 ts_tablet_manager.cc:1397] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:03:55.542454 11151 tablet_bootstrap.cc:492] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: No bootstrap required, opened a new log
I20260810 18:03:55.542629 11151 ts_tablet_manager.cc:1397] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260810 18:03:55.544222 11149 raft_consensus.cc:348] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:55.544564 11149 raft_consensus.cc:374] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:55.544646 11149 raft_consensus.cc:729] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe73cd93bb5d4cae839fb9f590709106, State: Initialized, Role: FOLLOWER
I20260810 18:03:55.545148 11149 consensus_queue.cc:260] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:55.546617 11149 ts_tablet_manager.cc:1428] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260810 18:03:55.546486 11151 raft_consensus.cc:348] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:55.546736 11151 raft_consensus.cc:374] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:55.546780 11151 raft_consensus.cc:729] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8451c71ffbea48ce91d5efd293c37643, State: Initialized, Role: FOLLOWER
I20260810 18:03:55.546808 11150 raft_consensus.cc:348] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:55.547017 10991 heartbeater.cc:503] Master 127.28.83.126:36225 was elected leader, sending a full tablet report...
I20260810 18:03:55.547080 11150 raft_consensus.cc:374] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:55.547070 11157 raft_consensus.cc:482] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:03:55.547178 11150 raft_consensus.cc:729] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe36616bec44765a949d3a96e04952a, State: Initialized, Role: FOLLOWER
I20260810 18:03:55.547204 11157 raft_consensus.cc:504] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:55.547555 11151 consensus_queue.cc:260] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:55.547705 11150 consensus_queue.cc:260] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:55.548569 11151 ts_tablet_manager.cc:1428] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:03:55.548703 11150 ts_tablet_manager.cc:1428] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:03:55.548935 11132 heartbeater.cc:503] Master 127.28.83.126:36225 was elected leader, sending a full tablet report...
I20260810 18:03:55.549067 10850 heartbeater.cc:503] Master 127.28.83.126:36225 was elected leader, sending a full tablet report...
I20260810 18:03:55.552484 11157 leader_election.cc:288] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625), ffe36616bec44765a949d3a96e04952a (127.28.83.65:36635)
I20260810 18:03:55.553553 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.548746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:36635 (local address 127.28.83.66:39569)
0810 18:03:55.549053 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:55.549069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:55.549152 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:55.549757 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:55.549760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:55.549778 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:55.549933 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.549943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.550936 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.550939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.552167 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.552179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.553053 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.553056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.553211 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.553235 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:03:55.553313 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:03:55.553387 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260810 18:03:55.553936 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.549300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:39569 (local address 127.28.83.65:36635)
0810 18:03:55.549431 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.549435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.549450 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:03:55.549495 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.549498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.549539 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.549624 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:55.550067 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.550800 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.552307 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.552918 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.553058 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.553108 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.553728 (+ 620us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.553801 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:03:55.554575 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31518501c1524a19a03bf18aa0c99d9f" candidate_uuid: "fe73cd93bb5d4cae839fb9f590709106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe36616bec44765a949d3a96e04952a" is_pre_election: true
I20260810 18:03:55.554852 10802 raft_consensus.cc:2454] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe73cd93bb5d4cae839fb9f590709106 in term 0.
I20260810 18:03:55.555315 10873 leader_election.cc:302] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fe73cd93bb5d4cae839fb9f590709106, ffe36616bec44765a949d3a96e04952a; no voters:
I20260810 18:03:55.555519 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.549294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:55247 (local address 127.28.83.67:46625)
0810 18:03:55.549435 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.549440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.550633 (+ 1193us) server_negotiation.cc:407] Connection header received
0810 18:03:55.550807 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.550810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.550859 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.550930 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:55.551351 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.552025 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.554268 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.554882 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.555004 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.555059 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.555340 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.555419 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:03:55.555598 11156 raft_consensus.cc:2781] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:03:55.555699 11156 raft_consensus.cc:482] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:03:55.555754 11156 raft_consensus.cc:3037] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:55.555925 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.548763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:46625 (local address 127.28.83.66:55247)
0810 18:03:55.550518 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:55.550530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:55.550694 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:55.551040 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:55.551042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:55.551081 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:55.551236 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.551242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.552136 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.552141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.553994 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:55.554089 (+ 95us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.555091 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.555095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:55.555193 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.555216 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:03:55.555718 (+ 502us) client_negotiation.cc:240] Negotiation successful
0810 18:03:55.555809 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1701,"thread_start_us":97,"threads_started":1}
I20260810 18:03:55.556527 11084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31518501c1524a19a03bf18aa0c99d9f" candidate_uuid: "fe73cd93bb5d4cae839fb9f590709106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8451c71ffbea48ce91d5efd293c37643" is_pre_election: true
I20260810 18:03:55.556806 11084 raft_consensus.cc:2454] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe73cd93bb5d4cae839fb9f590709106 in term 0.
I20260810 18:03:55.557190 11156 raft_consensus.cc:504] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:55.557610 11156 leader_election.cc:288] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [CANDIDATE]: Term 1 election: Requested vote from peers 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625), ffe36616bec44765a949d3a96e04952a (127.28.83.65:36635)
I20260810 18:03:55.557875 11084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31518501c1524a19a03bf18aa0c99d9f" candidate_uuid: "fe73cd93bb5d4cae839fb9f590709106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8451c71ffbea48ce91d5efd293c37643"
I20260810 18:03:55.557991 11084 raft_consensus.cc:3037] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:55.558014 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31518501c1524a19a03bf18aa0c99d9f" candidate_uuid: "fe73cd93bb5d4cae839fb9f590709106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe36616bec44765a949d3a96e04952a"
I20260810 18:03:55.558169 10802 raft_consensus.cc:3037] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:55.559046 11084 raft_consensus.cc:2454] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe73cd93bb5d4cae839fb9f590709106 in term 1.
I20260810 18:03:55.559262 10802 raft_consensus.cc:2454] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe73cd93bb5d4cae839fb9f590709106 in term 1.
I20260810 18:03:55.559549 10873 leader_election.cc:302] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fe73cd93bb5d4cae839fb9f590709106, ffe36616bec44765a949d3a96e04952a; no voters:
I20260810 18:03:55.559772 11156 raft_consensus.cc:2781] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:03:55.559873 11156 raft_consensus.cc:686] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 1 LEADER]: Becoming Leader. State: Replica: fe73cd93bb5d4cae839fb9f590709106, State: Running, Role: LEADER
I20260810 18:03:55.560190 11156 consensus_queue.cc:237] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:55.563381 10651 catalog_manager.cc:5495] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66). New cstate: current_term: 1 leader_uuid: "fe73cd93bb5d4cae839fb9f590709106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } health_report { overall_health: UNKNOWN } } }
I20260810 18:03:55.586762 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.583202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.28.83.66:41333)
0810 18:03:55.583314 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:55.583318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:55.583332 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:03:55.583409 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:55.583411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:55.583466 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:55.583553 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:03:55.583828 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.584345 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.585157 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:55.585372 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:55.585488 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:55.585881 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:03:55.586286 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:03:55.586380 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:55.586506 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 18:03:55.586558 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 18:03:55.592682 11084 raft_consensus.cc:1261] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [term 1 FOLLOWER]: Refusing update from remote peer fe73cd93bb5d4cae839fb9f590709106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:03:55.592715 10802 raft_consensus.cc:1261] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Refusing update from remote peer fe73cd93bb5d4cae839fb9f590709106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:03:55.593545 11156 consensus_queue.cc:1035] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:03:55.593793 11157 consensus_queue.cc:1035] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:03:55.602784 11167 mvcc.cc:205] Tried to move back new op lower bound from 7317033105779539968 to 7317033105657147392. Current Snapshot: MvccSnapshot[applied={T|T < 7317033105779539968}]
I20260810 18:03:55.603422 11165 mvcc.cc:205] Tried to move back new op lower bound from 7317033105779539968 to 7317033105657147392. Current Snapshot: MvccSnapshot[applied={T|T < 7317033105779539968}]
I20260810 18:03:55.610643 10653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:03:55.610848 10653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:03:55.613312 10653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Sending DeleteTablet for 3 replicas of tablet 31518501c1524a19a03bf18aa0c99d9f
I20260810 18:03:55.614156 10921 tablet_service.cc:1514] Processing DeleteTablet for tablet 31518501c1524a19a03bf18aa0c99d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:03:55 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:38610
I20260810 18:03:55.614378 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 10715
I20260810 18:03:55.614331 11063 tablet_service.cc:1514] Processing DeleteTablet for tablet 31518501c1524a19a03bf18aa0c99d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:03:55 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:54970
I20260810 18:03:55.615779 11176 tablet_replica.cc:330] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: stopping tablet replica
I20260810 18:03:55.616140 11176 raft_consensus.cc:2229] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:03:55.616598 11176 raft_consensus.cc:2258] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:03:55.618278 11176 ts_tablet_manager.cc:1905] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:03:55.620920 11176 ts_tablet_manager.cc:1918] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:03:55.621049 11176 log.cc:1196] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/31518501c1524a19a03bf18aa0c99d9f
I20260810 18:03:55.621502 11176 ts_tablet_manager.cc:1939] T 31518501c1524a19a03bf18aa0c99d9f P fe73cd93bb5d4cae839fb9f590709106: Deleting consensus metadata
I20260810 18:03:55.621665 11177 tablet_replica.cc:330] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: stopping tablet replica
I20260810 18:03:55.622052 11177 raft_consensus.cc:2229] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:03:55.622431 11177 raft_consensus.cc:2258] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:03:55.623100 10638 catalog_manager.cc:4841] TS fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333): tablet 31518501c1524a19a03bf18aa0c99d9f (table test-table [id=c2322eea7647445ab69060f1f66f8754]) successfully deleted
W20260810 18:03:55.624238 10635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:36635 (error 108)
I20260810 18:03:55.624850 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 10855
I20260810 18:03:55.626161 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:55.625010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:36635 (local address 127.0.0.1:56192)
0810 18:03:55.626022 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:55.626086 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":938}
W20260810 18:03:55.629709 10635 catalog_manager.cc:4584] TS ffe36616bec44765a949d3a96e04952a (127.28.83.65:36635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31518501c1524a19a03bf18aa0c99d9f: Network error: Client connection negotiation failed: client connection to 127.28.83.65:36635: connect: Connection refused (error 111)
I20260810 18:03:55.630311 11177 ts_tablet_manager.cc:1905] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:03:55.632555 11177 ts_tablet_manager.cc:1918] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:03:55.632642 11177 log.cc:1196] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/31518501c1524a19a03bf18aa0c99d9f
I20260810 18:03:55.632977 11177 ts_tablet_manager.cc:1939] T 31518501c1524a19a03bf18aa0c99d9f P 8451c71ffbea48ce91d5efd293c37643: Deleting consensus metadata
I20260810 18:03:55.634186 10636 catalog_manager.cc:4841] TS 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625): tablet 31518501c1524a19a03bf18aa0c99d9f (table test-table [id=c2322eea7647445ab69060f1f66f8754]) successfully deleted
I20260810 18:03:55.635380 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 10996
I20260810 18:03:55.641868 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 10619
I20260810 18:03:55.649273 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36225
--webserver_interface=127.28.83.126
--webserver_port=34403
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:55.759943 11179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:55.760272 11179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:55.760349 11179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:55.764405 11179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:03:55.764503 11179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:55.764542 11179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:55.764576 11179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:03:55.764607 11179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:03:55.769881 11179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=34403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:55.771009 11179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:55.772377 11179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:55.785843 11186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:55.785881 11188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:55.785888 11185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:55.786156 11179 server_base.cc:1034] running on GCE node
I20260810 18:03:55.786798 11179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:55.787981 11179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:55.789179 11179 hybrid_clock.cc:648] HybridClock initialized: now 1786385035789164 us; error 28 us; skew 500 ppm
Aug 10 18:03:55 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:55.792145 11179 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:03:55.793164 11179 webserver.cc:460] Webserver started at http://127.28.83.126:34403/ using document root <none> and password file <none>
I20260810 18:03:55.793843 11179 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:55.793957 11179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:55.797899 11179 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:03:55.800024 11195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.801283 11179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:03:55.801453 11179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "78d6054ec43d469d872a7c92b733fb97"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "d1164d3346585ef62449d0787544fc6e"
server_key_iv: "744eed96e0aa769e3946dc03263ba4ce"
server_key_version: "encryptionkey@0"
I20260810 18:03:55.802088 11179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:55.811278 11179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:55.814489 11179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:55.814706 11179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:55.821736 11179 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:36225
I20260810 18:03:55.822077 11250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:36225 every 8 connection(s)
I20260810 18:03:55.822876 11179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:03:55.825448 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 11179
I20260810 18:03:55.825945 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:36635
--local_ip_for_outbound_sockets=127.28.83.65
--tserver_master_addrs=127.28.83.126:36225
--webserver_port=43239
--webserver_interface=127.28.83.65
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:03:55.827910 11251 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:03:55.829666 11251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Bootstrap starting.
I20260810 18:03:55.847823 11251 log.cc:824] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:55.857193 11251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:03:55.857645 11251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Bootstrap complete.
I20260810 18:03:55.861999 11251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:55.862396 11251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d6054ec43d469d872a7c92b733fb97, State: Initialized, Role: FOLLOWER
I20260810 18:03:55.862883 11251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:55.862994 11251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:03:55.863094 11251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:03:55.863214 11251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:03:55.865135 11251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:55.865535 11251 leader_election.cc:302] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d6054ec43d469d872a7c92b733fb97; no voters:
I20260810 18:03:55.865878 11251 leader_election.cc:288] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:03:55.865965 11255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:03:55.866170 11255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 2 LEADER]: Becoming Leader. State: Replica: 78d6054ec43d469d872a7c92b733fb97, State: Running, Role: LEADER
I20260810 18:03:55.866458 11255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:55.867118 11251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:03:55.867405 11255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d6054ec43d469d872a7c92b733fb97. Latest consensus state: current_term: 2 leader_uuid: "78d6054ec43d469d872a7c92b733fb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } } }
I20260810 18:03:55.867524 11255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:55.867883 11259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:03:55.867909 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "78d6054ec43d469d872a7c92b733fb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } } }
I20260810 18:03:55.868136 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:55.872653 11259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:03:55.873435 11259 catalog_manager.cc:1259] Loaded cluster ID: 3a16610d2fca4f50afab55c7e78c32f7
I20260810 18:03:55.873497 11259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:03:55.874907 11259 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 18:03:55.880122 11259 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 18:03:55.885749 11259 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:03:55.886528 11259 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Loaded TSK: 0
I20260810 18:03:55.887691 11259 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:03:55.944854 11253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:55.945278 11253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:55.945348 11253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:55.951649 11253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:55.951741 11253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:55.951839 11253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:03:55.957316 11253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:36635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=43239
--tserver_master_addrs=127.28.83.126:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:55.958454 11253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:55.959471 11253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:55.969856 11278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:55.969911 11281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:55.969866 11279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:55.970701 11253 server_base.cc:1034] running on GCE node
I20260810 18:03:55.971143 11253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:55.971765 11253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:55.972946 11253 hybrid_clock.cc:648] HybridClock initialized: now 1786385035972928 us; error 44 us; skew 500 ppm
Aug 10 18:03:55 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:55.975911 11253 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:03:55.976876 11253 webserver.cc:460] Webserver started at http://127.28.83.65:43239/ using document root <none> and password file <none>
I20260810 18:03:55.977489 11253 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:55.977603 11253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:55.981384 11253 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:03:55.983807 11288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:55.985114 11253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:03:55.985258 11253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "ffe36616bec44765a949d3a96e04952a"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "b82ffd703d4b6461d7512a57b7a9453d"
server_key_iv: "7accbaba1fc73c905b41264192056ece"
server_key_version: "encryptionkey@0"
I20260810 18:03:55.985884 11253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:56.007526 11253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:56.011582 11253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:56.011839 11253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:56.012646 11253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:56.014003 11295 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:03:56.018326 11253 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:03:56.018419 11253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260810 18:03:56.018483 11253 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:03:56.020606 11253 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:03:56.020665 11253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 18:03:56.020998 11295 tablet_bootstrap.cc:492] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Bootstrap starting.
I20260810 18:03:56.031457 11253 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:36635
I20260810 18:03:56.031493 11407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:36635 every 8 connection(s)
I20260810 18:03:56.032436 11253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:03:56.033728 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 11253
I20260810 18:03:56.034243 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:41333
--local_ip_for_outbound_sockets=127.28.83.66
--tserver_master_addrs=127.28.83.126:36225
--webserver_port=36897
--webserver_interface=127.28.83.66
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:03:56.043212 11295 log.cc:824] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:03:56 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:56.052140 11411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.034792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:38329)
0810 18:03:56.035820 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.035879 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:56.037169 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:56.038510 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:56.038519 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:56.039010 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:56.039790 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.039813 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.041999 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.042002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.043302 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.043315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.044335 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.044339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.044528 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.045164 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:56.045181 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:56.046719 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:56.048923 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:56.048928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:56.048940 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:56.049204 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:56.049633 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:56.049637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:56.049639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:56.049782 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:56.050437 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:56.050444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:56.050697 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 18:03:56.050878 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 18:03:56.051217 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":179,"threads_started":1}
I20260810 18:03:56.052125 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.035014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:38329 (local address 127.28.83.126:36225)
0810 18:03:56.035502 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.035522 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.036024 (+ 502us) server_negotiation.cc:407] Connection header received
0810 18:03:56.037774 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.037785 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.038096 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.038364 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:56.040026 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.041812 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.043472 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.044175 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.044368 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.046942 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:56.046947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:56.046959 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:56.046987 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:56.048775 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.049328 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.049334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.049339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.049527 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.049882 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.049887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.049888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.050314 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:56.050449 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.050988 (+ 539us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.051281 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":121,"threads_started":1}
I20260810 18:03:56.052580 11295 tablet_bootstrap.cc:492] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:03:56.053009 11295 tablet_bootstrap.cc:492] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Bootstrap complete.
I20260810 18:03:56.054677 11295 ts_tablet_manager.cc:1397] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.011s
I20260810 18:03:56.054768 11408 heartbeater.cc:347] Connected to a master server at 127.28.83.126:36225
I20260810 18:03:56.055004 11408 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:56.055501 11408 heartbeater.cc:511] Master 127.28.83.126:36225 requested a full tablet report, sending...
I20260810 18:03:56.057801 11213 ts_manager.cc:194] Registered new tserver with Master: ffe36616bec44765a949d3a96e04952a (127.28.83.65:36635)
I20260810 18:03:56.057809 11295 raft_consensus.cc:348] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.058441 11295 raft_consensus.cc:729] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe36616bec44765a949d3a96e04952a, State: Initialized, Role: FOLLOWER
I20260810 18:03:56.058945 11295 consensus_queue.cc:260] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.059969 11295 ts_tablet_manager.cc:1428] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 18:03:56.061805 11213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:38329
I20260810 18:03:56.064229 11408 heartbeater.cc:503] Master 127.28.83.126:36225 was elected leader, sending a full tablet report...
Aug 10 18:03:56 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385035, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:03:56.072623 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.061000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.28.83.65:36635)
0810 18:03:56.062600 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.062609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.062628 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:03:56.062680 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.062684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.062879 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.063015 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:56.063781 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.064800 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.065756 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.066276 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.066405 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.068869 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:56.068877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:56.068886 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:56.068907 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:56.070653 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.071194 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.071198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.071200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.071275 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.071604 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.071608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.071609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.071849 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:56.071966 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.072331 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.072471 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1505,"thread_start_us":95,"threads_started":1}
I20260810 18:03:56.072623 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.060910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:36635 (local address 127.0.0.1:56194)
0810 18:03:56.061297 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.061352 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:56.061513 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:56.063045 (+ 1532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:56.063051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:56.063486 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:56.063629 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.063637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.064929 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.064931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.065601 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.065611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.066423 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.066426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.066533 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.067149 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:56.067161 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:56.068715 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:56.070765 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:56.070775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:56.070780 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:56.071077 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:56.071364 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:56.071367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:56.071369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:56.071466 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:56.071972 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:56.071977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:56.072103 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 18:03:56.072247 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 18:03:56.072439 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
I20260810 18:03:56.073825 11340 tablet_service.cc:1514] Processing DeleteTablet for tablet 31518501c1524a19a03bf18aa0c99d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:03:55 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:56194
I20260810 18:03:56.074862 11418 tablet_replica.cc:330] stopping tablet replica
I20260810 18:03:56.075196 11418 raft_consensus.cc:2229] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:03:56.075455 11418 raft_consensus.cc:2258] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:03:56.076499 11418 ts_tablet_manager.cc:1905] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:03:56.080262 11418 ts_tablet_manager.cc:1918] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:03:56.080348 11418 log.cc:1196] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/31518501c1524a19a03bf18aa0c99d9f
I20260810 18:03:56.080665 11418 ts_tablet_manager.cc:1939] T 31518501c1524a19a03bf18aa0c99d9f P ffe36616bec44765a949d3a96e04952a: Deleting consensus metadata
I20260810 18:03:56.082365 11196 catalog_manager.cc:4841] TS ffe36616bec44765a949d3a96e04952a (127.28.83.65:36635): tablet 31518501c1524a19a03bf18aa0c99d9f (table test-table [id=c2322eea7647445ab69060f1f66f8754]) successfully deleted
W20260810 18:03:56.158463 11413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:56.158761 11413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:56.158839 11413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:56.162911 11413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:56.162997 11413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:56.163093 11413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:03:56.168358 11413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:41333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=36897
--tserver_master_addrs=127.28.83.126:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:56.169399 11413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:56.170475 11413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:56.179838 11424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:56.179848 11423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:56.179847 11426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:56.180198 11413 server_base.cc:1034] running on GCE node
I20260810 18:03:56.180616 11413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:56.181259 11413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:56.182446 11413 hybrid_clock.cc:648] HybridClock initialized: now 1786385036182421 us; error 34 us; skew 500 ppm
Aug 10 18:03:56 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:56.185465 11413 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:03:56.186448 11413 webserver.cc:460] Webserver started at http://127.28.83.66:36897/ using document root <none> and password file <none>
I20260810 18:03:56.187100 11413 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:56.187206 11413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:56.191126 11413 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:03:56.193148 11433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:56.194419 11413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:03:56.194561 11413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "fe73cd93bb5d4cae839fb9f590709106"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "6d631932a5587d5bb8d8f0dbd16d736a"
server_key_iv: "1fb3f6cddd35fbd51042d6bafd139b89"
server_key_version: "encryptionkey@0"
I20260810 18:03:56.195139 11413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:56.206879 11413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:56.210178 11413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:56.210381 11413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:56.210959 11413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:56.211771 11413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:56.211843 11413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:56.211926 11413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:56.211963 11413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:56.220925 11413 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:41333
I20260810 18:03:56.220960 11557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:41333 every 8 connection(s)
I20260810 18:03:56.221930 11413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:03:56.222960 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 11413
I20260810 18:03:56.223368 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:46625
--local_ip_for_outbound_sockets=127.28.83.67
--tserver_master_addrs=127.28.83.126:36225
--webserver_port=36021
--webserver_interface=127.28.83.67
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:03:56 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:56.235858 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.223724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:38697 (local address 127.28.83.126:36225)
0810 18:03:56.223911 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.223918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.224834 (+ 916us) server_negotiation.cc:407] Connection header received
0810 18:03:56.226051 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.226054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.226110 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.226190 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:56.227296 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.227905 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.228995 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.229240 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.229376 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.232120 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:56.232124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:56.232126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:56.232153 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:56.233741 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.234294 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.234297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.234299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.234383 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.234729 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.234734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.234736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.234915 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:56.235027 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.235524 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.235677 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:03:56.236647 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.224137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:38697)
0810 18:03:56.224671 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.224729 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:56.225844 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:56.226347 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:56.226354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:56.226689 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:56.227148 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.227161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.228037 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.228044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.228866 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.228875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.229418 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.229422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.229629 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.230223 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:56.230238 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:56.231950 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:56.233892 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:56.233898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:56.233912 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:56.234184 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:56.234480 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:56.234484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:56.234486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:56.234631 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:56.235025 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:56.235032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:56.235292 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 18:03:56.235483 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 18:03:56.235806 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":119,"threads_started":1}
I20260810 18:03:56.237942 11558 heartbeater.cc:347] Connected to a master server at 127.28.83.126:36225
I20260810 18:03:56.238193 11558 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:56.238740 11558 heartbeater.cc:511] Master 127.28.83.126:36225 requested a full tablet report, sending...
I20260810 18:03:56.239899 11213 ts_manager.cc:194] Registered new tserver with Master: fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333)
I20260810 18:03:56.240651 11213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:38697
W20260810 18:03:56.336789 11561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:56.337055 11561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:56.337100 11561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:56.341462 11561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:56.341532 11561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:56.341591 11561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:03:56.347338 11561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:46625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=36021
--tserver_master_addrs=127.28.83.126:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:56.348361 11561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:56.349342 11561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:56.359283 11569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:56.359277 11566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:56.359362 11561 server_base.cc:1034] running on GCE node
W20260810 18:03:56.359277 11567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:56.360014 11561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:56.360656 11561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:56.361851 11561 hybrid_clock.cc:648] HybridClock initialized: now 1786385036361822 us; error 41 us; skew 500 ppm
Aug 10 18:03:56 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:56.364670 11561 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:03:56.365803 11561 webserver.cc:460] Webserver started at http://127.28.83.67:36021/ using document root <none> and password file <none>
I20260810 18:03:56.366339 11561 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:56.366415 11561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:56.370378 11561 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:03:56.372576 11577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:56.373965 11561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:03:56.374087 11561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "8451c71ffbea48ce91d5efd293c37643"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "30ad38ededfc08bc69be84b6cffe2a42"
server_key_iv: "c9c16924f16c364e38b8373f05e47e01"
server_key_version: "encryptionkey@0"
I20260810 18:03:56.374598 11561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:56.388785 11561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:56.392108 11561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:56.392326 11561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:56.392980 11561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:56.393896 11561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:56.393950 11561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:56.393990 11561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:56.394035 11561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:56.403082 11561 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:46625
I20260810 18:03:56.403105 11701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:46625 every 8 connection(s)
I20260810 18:03:56.404037 11561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:03:56.410413 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 11561
Aug 10 18:03:56 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:03:56.418124 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.406130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:48063 (local address 127.28.83.126:36225)
0810 18:03:56.406279 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.406283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.407128 (+ 845us) server_negotiation.cc:407] Connection header received
0810 18:03:56.408354 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.408357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.408404 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.408480 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:56.409692 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.410331 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.411804 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.412022 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.412136 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.414738 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:56.414741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:56.414743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:56.414777 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:56.416075 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.416580 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.416583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.416584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.416684 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.417030 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.417034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.417035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.417279 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:56.417390 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.417895 (+ 505us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.417987 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:03:56.419064 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.406538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:48063)
0810 18:03:56.406971 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.407011 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:56.408170 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:56.408638 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:56.408644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:56.409009 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:56.409560 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.409575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.410457 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.410460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.411672 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.411682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.412174 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.412178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.412351 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.412939 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:56.412954 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:56.414558 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:56.416195 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:56.416200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:56.416212 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:56.416489 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:56.416797 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:56.416799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:56.416800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:56.416929 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:56.417397 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:56.417402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:56.417669 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 18:03:56.417824 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 18:03:56.418168 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260810 18:03:56.419589 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.411120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.28.83.126:36225)
0810 18:03:56.411453 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.411456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.411541 (+ 85us) server_negotiation.cc:407] Connection header received
0810 18:03:56.411660 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.411662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.411700 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.411809 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:56.412515 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.413140 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.414380 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.414603 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.414713 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.416236 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:56.416239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:56.416240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:56.416260 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:56.417781 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.418438 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.418441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.418442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.418510 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:56.418791 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:56.418793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:56.418795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:56.418935 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:56.419066 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.419345 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.419452 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"spinlock_wait_cycles":3840,"thread_start_us":93,"threads_started":1}
I20260810 18:03:56.420194 11702 heartbeater.cc:347] Connected to a master server at 127.28.83.126:36225
I20260810 18:03:56.420441 11702 heartbeater.cc:464] Registering TS with master...
I20260810 18:03:56.420923 11702 heartbeater.cc:511] Master 127.28.83.126:36225 requested a full tablet report, sending...
I20260810 18:03:56.421898 11213 ts_manager.cc:194] Registered new tserver with Master: 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625)
I20260810 18:03:56.422441 11213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:48063
I20260810 18:03:56.433379 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:03:56.438041 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.434155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.28.83.126:36225)
0810 18:03:56.434308 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.434311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.434388 (+ 77us) server_negotiation.cc:407] Connection header received
0810 18:03:56.434528 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.434530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.434628 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.434721 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:03:56.434927 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.435589 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.436537 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.436764 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.436849 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.437137 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:03:56.437618 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:03:56.437725 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.437818 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.437868 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 18:03:56.438728 11213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:03:56.442145 11213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:03:56.450309 11340 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1082e3e4cf455e92f5987fe0caf3e2 (DEFAULT_TABLE table=test-table [id=04fe45ab04c84f64a8386ef911ed461f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:03:56.451071 11340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1082e3e4cf455e92f5987fe0caf3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:56.454106 11295 tablet_bootstrap.cc:492] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Bootstrap starting.
I20260810 18:03:56.455081 11295 tablet_bootstrap.cc:654] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:56.455962 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.449048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41333 (local address 127.0.0.1:40478)
0810 18:03:56.449180 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.449197 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:56.449282 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:56.450266 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:56.450269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:56.450290 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:56.450433 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.450442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.452777 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.452781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.454029 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.454040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.455471 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.455475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.455566 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.455660 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 18:03:56.455749 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 18:03:56.455830 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":68,"spinlock_wait_cycles":7296}
I20260810 18:03:56.456022 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.449228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.28.83.66:41333)
0810 18:03:56.449690 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.449705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.449730 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:03:56.449818 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.449826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.450062 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.450778 (+ 716us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:56.451148 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.452592 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.454493 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.455335 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.455471 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.455608 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.455795 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.455886 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":195,"threads_started":1}
I20260810 18:03:56.456135 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.449336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:46625 (local address 127.0.0.1:43278)
0810 18:03:56.450212 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.450228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:56.450332 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:56.451057 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:56.451060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:56.451079 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:56.451177 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.451186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.452833 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.452836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.453972 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.453986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.455514 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.455519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.455691 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.455731 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 18:03:56.455813 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:03:56.455894 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":63,"spinlock_wait_cycles":768,"thread_start_us":128,"threads_started":1}
I20260810 18:03:56.456326 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.449797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.28.83.67:46625)
0810 18:03:56.450292 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.450305 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.450331 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:03:56.450458 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.450467 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.450687 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.450845 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:56.451334 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.452656 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.454149 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.455088 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.455215 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.455417 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.456033 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.456150 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":179,"threads_started":1}
I20260810 18:03:56.456866 11295 tablet_bootstrap.cc:492] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: No bootstrap required, opened a new log
I20260810 18:03:56.456977 11295 ts_tablet_manager.cc:1397] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 18:03:56.457491 11295 raft_consensus.cc:348] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.457772 11295 raft_consensus.cc:374] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:56.457819 11295 raft_consensus.cc:729] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe36616bec44765a949d3a96e04952a, State: Initialized, Role: FOLLOWER
I20260810 18:03:56.457968 11295 consensus_queue.cc:260] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.458330 11295 ts_tablet_manager.cc:1428] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:03:56.459394 11485 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1082e3e4cf455e92f5987fe0caf3e2 (DEFAULT_TABLE table=test-table [id=04fe45ab04c84f64a8386ef911ed461f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:03:56.459453 11631 tablet_service.cc:1467] Processing CreateTablet for tablet 1b1082e3e4cf455e92f5987fe0caf3e2 (DEFAULT_TABLE table=test-table [id=04fe45ab04c84f64a8386ef911ed461f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:03:56.460312 11485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1082e3e4cf455e92f5987fe0caf3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:56.460510 11631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b1082e3e4cf455e92f5987fe0caf3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:03:56.467301 11415 raft_consensus.cc:482] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:03:56.467478 11415 raft_consensus.cc:504] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.467598 11716 tablet_bootstrap.cc:492] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: Bootstrap starting.
I20260810 18:03:56.468672 11415 leader_election.cc:288] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333), 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625)
I20260810 18:03:56.469225 11717 tablet_bootstrap.cc:492] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: Bootstrap starting.
I20260810 18:03:56.470563 11716 tablet_bootstrap.cc:654] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:56.471441 11716 log.cc:824] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:56.472009 11717 tablet_bootstrap.cc:654] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: Neither blocks nor log segments found. Creating new log.
I20260810 18:03:56.472906 11717 log.cc:824] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:56.473557 11411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.468833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41333 (local address 127.28.83.65:34717)
0810 18:03:56.469059 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.469074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:56.469177 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:56.469808 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:56.469811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:56.469829 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:56.469936 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.469942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.470970 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.470973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.472126 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.472138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.473093 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.473099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.473211 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.473259 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 18:03:56.473334 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:03:56.473431 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260810 18:03:56.474011 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.469357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:34717 (local address 127.28.83.66:41333)
0810 18:03:56.469517 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.469522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.469536 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:03:56.469589 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.469592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.469634 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.469703 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:56.470067 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.470837 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.472261 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.472819 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.472967 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.473025 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.473834 (+ 809us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.473909 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 18:03:56.474152 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.469265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:37039 (local address 127.28.83.67:46625)
0810 18:03:56.469398 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.469421 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.469704 (+ 283us) server_negotiation.cc:407] Connection header received
0810 18:03:56.469877 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.469880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.469923 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.469968 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:03:56.470408 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.471069 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.472589 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.473206 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.473320 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.473373 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.473915 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.474001 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:03:56.474428 11716 tablet_bootstrap.cc:492] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: No bootstrap required, opened a new log
I20260810 18:03:56.474588 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.468872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:46625 (local address 127.28.83.65:37039)
0810 18:03:56.469620 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.469634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:56.469767 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:56.470114 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:56.470118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:56.470138 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:03:56.470279 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.470288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.471223 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.471225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.472445 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:56.472456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.473613 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.473620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:56.473755 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.473785 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:03:56.474343 (+ 558us) client_negotiation.cc:240] Negotiation successful
0810 18:03:56.474432 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":116,"threads_started":1}
I20260810 18:03:56.474637 11716 ts_tablet_manager.cc:1397] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:03:56.475612 11717 tablet_bootstrap.cc:492] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: No bootstrap required, opened a new log
I20260810 18:03:56.475746 11717 ts_tablet_manager.cc:1397] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:03:56.474529 11506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1082e3e4cf455e92f5987fe0caf3e2" candidate_uuid: "ffe36616bec44765a949d3a96e04952a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe73cd93bb5d4cae839fb9f590709106" is_pre_election: true
W20260810 18:03:56.476799 11292 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:03:56.477840 11717 raft_consensus.cc:348] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.478092 11717 raft_consensus.cc:374] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:56.478132 11717 raft_consensus.cc:729] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe73cd93bb5d4cae839fb9f590709106, State: Initialized, Role: FOLLOWER
I20260810 18:03:56.478155 11716 raft_consensus.cc:348] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.478382 11716 raft_consensus.cc:374] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:03:56.478418 11716 raft_consensus.cc:729] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8451c71ffbea48ce91d5efd293c37643, State: Initialized, Role: FOLLOWER
I20260810 18:03:56.478582 11717 consensus_queue.cc:260] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.479254 11716 consensus_queue.cc:260] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.479363 11558 heartbeater.cc:503] Master 127.28.83.126:36225 was elected leader, sending a full tablet report...
I20260810 18:03:56.479665 11717 ts_tablet_manager.cc:1428] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 18:03:56.479861 11702 heartbeater.cc:503] Master 127.28.83.126:36225 was elected leader, sending a full tablet report...
I20260810 18:03:56.480105 11653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1082e3e4cf455e92f5987fe0caf3e2" candidate_uuid: "ffe36616bec44765a949d3a96e04952a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8451c71ffbea48ce91d5efd293c37643" is_pre_election: true
I20260810 18:03:56.480358 11653 raft_consensus.cc:2454] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffe36616bec44765a949d3a96e04952a in term 0.
I20260810 18:03:56.480935 11290 leader_election.cc:302] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8451c71ffbea48ce91d5efd293c37643, ffe36616bec44765a949d3a96e04952a; no voters: fe73cd93bb5d4cae839fb9f590709106
I20260810 18:03:56.481256 11415 raft_consensus.cc:2781] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:03:56.481351 11415 raft_consensus.cc:482] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:03:56.481395 11415 raft_consensus.cc:3037] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:56.482508 11716 ts_tablet_manager.cc:1428] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:03:56.482797 11415 raft_consensus.cc:504] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.483196 11415 leader_election.cc:288] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 1 election: Requested vote from peers fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333), 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625)
I20260810 18:03:56.483603 11506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1082e3e4cf455e92f5987fe0caf3e2" candidate_uuid: "ffe36616bec44765a949d3a96e04952a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe73cd93bb5d4cae839fb9f590709106"
I20260810 18:03:56.483649 11653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b1082e3e4cf455e92f5987fe0caf3e2" candidate_uuid: "ffe36616bec44765a949d3a96e04952a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8451c71ffbea48ce91d5efd293c37643"
I20260810 18:03:56.483717 11506 raft_consensus.cc:3037] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:56.483757 11653 raft_consensus.cc:3037] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:03:56.484771 11653 raft_consensus.cc:2454] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe36616bec44765a949d3a96e04952a in term 1.
I20260810 18:03:56.484774 11506 raft_consensus.cc:2454] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe36616bec44765a949d3a96e04952a in term 1.
I20260810 18:03:56.485035 11290 leader_election.cc:302] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8451c71ffbea48ce91d5efd293c37643, ffe36616bec44765a949d3a96e04952a; no voters:
I20260810 18:03:56.485227 11415 raft_consensus.cc:2781] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:03:56.485472 11415 raft_consensus.cc:686] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 LEADER]: Becoming Leader. State: Replica: ffe36616bec44765a949d3a96e04952a, State: Running, Role: LEADER
I20260810 18:03:56.485720 11415 consensus_queue.cc:237] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:56.489073 11212 catalog_manager.cc:5495] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a reported cstate change: term changed from 0 to 1, leader changed from <none> to ffe36616bec44765a949d3a96e04952a (127.28.83.65). New cstate: current_term: 1 leader_uuid: "ffe36616bec44765a949d3a96e04952a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } health_report { overall_health: HEALTHY } } }
I20260810 18:03:56.507916 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.504347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.28.83.65:36635)
0810 18:03:56.504493 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:56.504498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:56.504513 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:03:56.504645 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:56.504648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:56.504702 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:56.504828 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:03:56.505072 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.505795 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.506643 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:56.506868 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:56.506948 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:56.507170 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:03:56.507613 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:03:56.507699 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:56.507757 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 18:03:56.507807 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:03:56.511996 11506 raft_consensus.cc:1261] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106 [term 1 FOLLOWER]: Refusing update from remote peer ffe36616bec44765a949d3a96e04952a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:03:56.512041 11653 raft_consensus.cc:1261] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [term 1 FOLLOWER]: Refusing update from remote peer ffe36616bec44765a949d3a96e04952a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:03:56.512722 11724 consensus_queue.cc:1035] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:03:56.512928 11415 consensus_queue.cc:1035] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:03:56.520123 11727 mvcc.cc:205] Tried to move back new op lower bound from 7317033109546430464 to 7317033109449187328. Current Snapshot: MvccSnapshot[applied={T|T < 7317033109546430464}]
I20260810 18:03:56.522146 11728 mvcc.cc:205] Tried to move back new op lower bound from 7317033109546430464 to 7317033109449187328. Current Snapshot: MvccSnapshot[applied={T|T < 7317033109546430464}]
I20260810 18:03:56.524168 11729 mvcc.cc:205] Tried to move back new op lower bound from 7317033109546430464 to 7317033109449187328. Current Snapshot: MvccSnapshot[applied={T|T < 7317033109546430464}]
I20260810 18:03:56.528640 11213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:03:56.528806 11213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:03:56.531023 11213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Sending DeleteTablet for 3 replicas of tablet 1b1082e3e4cf455e92f5987fe0caf3e2
I20260810 18:03:56.531711 11485 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1082e3e4cf455e92f5987fe0caf3e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:03:56 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:40478
I20260810 18:03:56.531767 11631 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1082e3e4cf455e92f5987fe0caf3e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:03:56 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:43278
I20260810 18:03:56.531831 11340 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b1082e3e4cf455e92f5987fe0caf3e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:03:56 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:56194
I20260810 18:03:56.531992 11418 tablet_replica.cc:330] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: stopping tablet replica
I20260810 18:03:56.532141 11418 raft_consensus.cc:2229] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:03:56.532306 11736 tablet_replica.cc:330] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: stopping tablet replica
I20260810 18:03:56.532344 11737 tablet_replica.cc:330] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: stopping tablet replica
I20260810 18:03:56.532408 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 11253
I20260810 18:03:56.532414 11418 raft_consensus.cc:2258] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:03:56.533951 11736 raft_consensus.cc:2229] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:03:56.534338 11736 raft_consensus.cc:2258] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:03:56.534711 11737 raft_consensus.cc:2229] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:03:56.535002 11737 raft_consensus.cc:2258] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:03:56.535980 11736 ts_tablet_manager.cc:1905] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:03:56.536273 11737 ts_tablet_manager.cc:1905] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:03:56.538813 11736 ts_tablet_manager.cc:1918] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:03:56.538813 11737 ts_tablet_manager.cc:1918] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:03:56.538936 11737 log.cc:1196] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/1b1082e3e4cf455e92f5987fe0caf3e2
I20260810 18:03:56.538936 11736 log.cc:1196] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/1b1082e3e4cf455e92f5987fe0caf3e2
I20260810 18:03:56.539466 11737 ts_tablet_manager.cc:1939] T 1b1082e3e4cf455e92f5987fe0caf3e2 P 8451c71ffbea48ce91d5efd293c37643: Deleting consensus metadata
I20260810 18:03:56.539698 11736 ts_tablet_manager.cc:1939] T 1b1082e3e4cf455e92f5987fe0caf3e2 P fe73cd93bb5d4cae839fb9f590709106: Deleting consensus metadata
W20260810 18:03:56.540628 11196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:36635 (error 108)
I20260810 18:03:56.540752 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 11413
I20260810 18:03:56.541328 11197 catalog_manager.cc:4841] TS 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625): tablet 1b1082e3e4cf455e92f5987fe0caf3e2 (table test-table [id=04fe45ab04c84f64a8386ef911ed461f]) successfully deleted
I20260810 18:03:56.542488 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.542096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:36635 (local address 127.0.0.1:39568)
0810 18:03:56.542315 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.542391 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260810 18:03:56.543720 11196 catalog_manager.cc:4584] TS ffe36616bec44765a949d3a96e04952a (127.28.83.65:36635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b1082e3e4cf455e92f5987fe0caf3e2: Network error: Client connection negotiation failed: client connection to 127.28.83.65:36635: connect: Connection refused (error 111)
I20260810 18:03:56.548277 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 11561
I20260810 18:03:56.549815 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:56.548490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41333 (local address 127.0.0.1:40482)
0810 18:03:56.549646 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.549714 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.66:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1051}
I20260810 18:03:56.554733 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 11179
I20260810 18:03:56.562216 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36225
--webserver_interface=127.28.83.126
--webserver_port=34403
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:56.674432 11739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:56.674690 11739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:56.674731 11739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:56.678917 11739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:03:56.678979 11739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:56.678998 11739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:56.679015 11739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:03:56.679030 11739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:03:56.684852 11739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=34403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:56.685971 11739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:56.686982 11739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:56.695820 11745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:56.695815 11748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:56.695796 11739 server_base.cc:1034] running on GCE node
W20260810 18:03:56.695778 11746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:56.696602 11739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:56.697664 11739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:56.698822 11739 hybrid_clock.cc:648] HybridClock initialized: now 1786385036698805 us; error 31 us; skew 500 ppm
Aug 10 18:03:56 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:56.701606 11739 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:03:56.702401 11739 webserver.cc:460] Webserver started at http://127.28.83.126:34403/ using document root <none> and password file <none>
I20260810 18:03:56.702906 11739 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:56.702965 11739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:56.706678 11739 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:03:56.708827 11755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:56.710117 11739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:03:56.710218 11739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "78d6054ec43d469d872a7c92b733fb97"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "d1164d3346585ef62449d0787544fc6e"
server_key_iv: "744eed96e0aa769e3946dc03263ba4ce"
server_key_version: "encryptionkey@0"
I20260810 18:03:56.710767 11739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:56.749532 11739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:56.753015 11739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:56.753201 11739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:56.760471 11739 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:36225
I20260810 18:03:56.760798 11814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:36225 every 8 connection(s)
I20260810 18:03:56.761669 11739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:03:56.766136 11815 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:03:56.767830 11815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Bootstrap starting.
I20260810 18:03:56.769356 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 11739
I20260810 18:03:56.769841 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:36635
--local_ip_for_outbound_sockets=127.28.83.65
--tserver_master_addrs=127.28.83.126:36225
--webserver_port=43239
--webserver_interface=127.28.83.65
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:03:56.786221 11815 log.cc:824] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:56.797777 11815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:03:56.798197 11815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97: Bootstrap complete.
I20260810 18:03:56.802445 11815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:56.802863 11815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d6054ec43d469d872a7c92b733fb97, State: Initialized, Role: FOLLOWER
I20260810 18:03:56.803335 11815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:56.803479 11815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:03:56.803567 11815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:03:56.803694 11815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:03:56.805755 11815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:56.806087 11815 leader_election.cc:302] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d6054ec43d469d872a7c92b733fb97; no voters:
I20260810 18:03:56.806434 11815 leader_election.cc:288] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 18:03:56.806524 11819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:03:56.806707 11819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [term 3 LEADER]: Becoming Leader. State: Replica: 78d6054ec43d469d872a7c92b733fb97, State: Running, Role: LEADER
I20260810 18:03:56.807003 11819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } }
I20260810 18:03:56.807686 11815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:03:56.807943 11819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d6054ec43d469d872a7c92b733fb97. Latest consensus state: current_term: 3 leader_uuid: "78d6054ec43d469d872a7c92b733fb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } } }
I20260810 18:03:56.808060 11819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:56.808470 11820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "78d6054ec43d469d872a7c92b733fb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d6054ec43d469d872a7c92b733fb97" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 36225 } } }
I20260810 18:03:56.808645 11820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d6054ec43d469d872a7c92b733fb97 [sys.catalog]: This master's current role is: LEADER
I20260810 18:03:56.810171 11823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:03:56.814258 11823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:03:56.814872 11823 catalog_manager.cc:1259] Loaded cluster ID: 3a16610d2fca4f50afab55c7e78c32f7
I20260810 18:03:56.814950 11823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 18:03:56.815922 11823 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc5acd4e9ad google::LogMessage::Fail() at ??:0
@ 0x7fc5acd5188a google::LogMessage::SendToLog() at ??:0
@ 0x7fc5acd4e4de google::LogMessage::Flush() at ??:0
@ 0x7fc5acd4fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc5b7691fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc5b7692d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc5b768f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc5b76d3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc5aefa03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc5aefa0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc5aefa23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc5aef91512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc5b05d56db start_thread at ??:0
@ 0x7fc5ab84c61f clone at ??:0
W20260810 18:03:56.903821 11817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:56.904129 11817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:56.904239 11817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:56.908429 11817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:56.908517 11817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:56.908618 11817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:03:56.914098 11817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:36635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=43239
--tserver_master_addrs=127.28.83.126:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:56.915239 11817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:56.916332 11817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:56.925822 11861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:56.925828 11859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:56.925828 11858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:56.926337 11817 server_base.cc:1034] running on GCE node
I20260810 18:03:56.926754 11817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:56.927455 11817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:56.928633 11817 hybrid_clock.cc:648] HybridClock initialized: now 1786385036928607 us; error 43 us; skew 500 ppm
Aug 10 18:03:56 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:56.931581 11817 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:03:56.932483 11817 webserver.cc:460] Webserver started at http://127.28.83.65:43239/ using document root <none> and password file <none>
I20260810 18:03:56.933035 11817 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:56.933102 11817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:56.937072 11817 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260810 18:03:56.939391 11869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:56.940729 11817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:03:56.940866 11817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "ffe36616bec44765a949d3a96e04952a"
format_stamp: "Formatted at 2026-08-10 18:03:54 on dist-test-slave-srgv"
server_key: "b82ffd703d4b6461d7512a57b7a9453d"
server_key_iv: "7accbaba1fc73c905b41264192056ece"
server_key_version: "encryptionkey@0"
I20260810 18:03:56.941458 11817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:56.962934 11817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:56.966452 11817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:56.966717 11817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:56.967397 11817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:56.968549 11876 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:03:56.972826 11817 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:03:56.972901 11817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 18:03:56.972954 11817 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:03:56.974753 11817 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:03:56.974817 11817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260810 18:03:56.975069 11876 tablet_bootstrap.cc:492] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Bootstrap starting.
I20260810 18:03:56.985929 11817 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:36635
I20260810 18:03:56.987351 11989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:36635 every 8 connection(s)
I20260810 18:03:56.987371 11817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:03:56.997448 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 11817
I20260810 18:03:56.997979 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:41333
--local_ip_for_outbound_sockets=127.28.83.66
--tserver_master_addrs=127.28.83.126:36225
--webserver_port=36897
--webserver_interface=127.28.83.66
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:03:56.998389 11876 log.cc:824] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:03:57.006922 11876 tablet_bootstrap.cc:492] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:03:57.007395 11876 tablet_bootstrap.cc:492] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Bootstrap complete.
I20260810 18:03:57.009092 11876 ts_tablet_manager.cc:1397] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.004s
I20260810 18:03:57.013449 11876 raft_consensus.cc:348] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:57.014140 11876 raft_consensus.cc:729] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe36616bec44765a949d3a96e04952a, State: Initialized, Role: FOLLOWER
I20260810 18:03:57.014734 11876 consensus_queue.cc:260] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:57.015825 11876 ts_tablet_manager.cc:1428] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a: Time spent starting tablet: real 0.007s user 0.010s sys 0.000s
W20260810 18:03:57.044377 11992 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:03:56.990075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:56851)
0810 18:03:56.990820 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:56.990884 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:56.991917 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:57.043563 (+ 51646us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":186,"threads_started":1}
W20260810 18:03:57.044870 11990 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:36225 (0 consecutive failures): Network error: Failed to ping master at 127.28.83.126:36225: Client connection negotiation failed: client connection to 127.28.83.126:36225: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 18:03:57.045718 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:57.045476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:33241)
0810 18:03:57.045601 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:57.045664 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 18:03:57.046384 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:57.046189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:39391)
0810 18:03:57.046300 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:57.046341 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:03:57.046624 11990 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:03:57.119392 11994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:57.119645 11994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:57.119688 11994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:57.123692 11994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:57.123745 11994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:57.123849 11994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:03:57.129056 11994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:41333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=36897
--tserver_master_addrs=127.28.83.126:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:57.130070 11994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:57.131078 11994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:57.140424 12001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:57.140426 12000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:57.140419 12003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:57.140641 11994 server_base.cc:1034] running on GCE node
I20260810 18:03:57.141062 11994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:57.141758 11994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:57.142949 11994 hybrid_clock.cc:648] HybridClock initialized: now 1786385037142939 us; error 31 us; skew 500 ppm
Aug 10 18:03:57 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385037, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:57.146073 11994 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:03:57.147145 11994 webserver.cc:460] Webserver started at http://127.28.83.66:36897/ using document root <none> and password file <none>
I20260810 18:03:57.147653 11994 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:57.147713 11994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:57.151434 11994 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:03:57.153399 12010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:57.154628 11994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:03:57.154760 11994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "fe73cd93bb5d4cae839fb9f590709106"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "6d631932a5587d5bb8d8f0dbd16d736a"
server_key_iv: "1fb3f6cddd35fbd51042d6bafd139b89"
server_key_version: "encryptionkey@0"
I20260810 18:03:57.155283 11994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:57.169545 11994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:57.172256 11994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:57.172454 11994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:57.173017 11994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:57.173889 11994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:57.173938 11994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:57.174006 11994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:57.174044 11994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:57.183564 11994 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:41333
I20260810 18:03:57.183593 12129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:41333 every 8 connection(s)
I20260810 18:03:57.184612 11994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:03:57.188349 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:57.186750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:33165)
0810 18:03:57.187205 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:57.187649 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
W20260810 18:03:57.188908 12130 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:36225 (0 consecutive failures): Network error: Failed to ping master at 127.28.83.126:36225: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
I20260810 18:03:57.189496 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:57.189279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:33057)
0810 18:03:57.189379 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:57.189442 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 18:03:57.190095 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:57.189905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:58225)
0810 18:03:57.190005 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:57.190045 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:03:57.190392 12130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:03:57.190591 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 11994
I20260810 18:03:57.190934 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:46625
--local_ip_for_outbound_sockets=127.28.83.67
--tserver_master_addrs=127.28.83.126:36225
--webserver_port=36021
--webserver_interface=127.28.83.67
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:03:57.301069 12133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:03:57.301308 12133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:03:57.301349 12133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:03:57.305223 12133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:03:57.305284 12133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:03:57.305377 12133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:03:57.310604 12133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:46625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=36021
--tserver_master_addrs=127.28.83.126:36225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:03:57.311631 12133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:03:57.312661 12133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:03:57.322113 12138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:03:57.322113 12139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:57.322232 12133 server_base.cc:1034] running on GCE node
W20260810 18:03:57.322180 12141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:03:57.322922 12133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:03:57.323558 12133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:03:57.324753 12133 hybrid_clock.cc:648] HybridClock initialized: now 1786385037324719 us; error 46 us; skew 500 ppm
Aug 10 18:03:57 dist-test-slave-srgv krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385037, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:03:57.327812 12133 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:03:57.328679 12133 webserver.cc:460] Webserver started at http://127.28.83.67:36021/ using document root <none> and password file <none>
I20260810 18:03:57.329257 12133 fs_manager.cc:359] Metadata directory not provided
I20260810 18:03:57.329357 12133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:03:57.333096 12133 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:03:57.335165 12148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:57.336414 12133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 18:03:57.336542 12133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "8451c71ffbea48ce91d5efd293c37643"
format_stamp: "Formatted at 2026-08-10 18:03:55 on dist-test-slave-srgv"
server_key: "30ad38ededfc08bc69be84b6cffe2a42"
server_key_iv: "c9c16924f16c364e38b8373f05e47e01"
server_key_version: "encryptionkey@0"
I20260810 18:03:57.337203 12133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:03:57.353241 12133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:03:57.356659 12133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:03:57.356817 12133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:03:57.357352 12133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:03:57.358278 12133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:03:57.358327 12133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:57.358368 12133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:03:57.358390 12133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:03:57.367460 12133 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:46625
I20260810 18:03:57.367489 12265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:46625 every 8 connection(s)
I20260810 18:03:57.368459 12133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:03:57.372068 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:57.370541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:38139)
0810 18:03:57.370993 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:57.371397 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
W20260810 18:03:57.372481 12266 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:36225 (0 consecutive failures): Network error: Failed to ping master at 127.28.83.126:36225: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
I20260810 18:03:57.373198 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:57.372965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:40843)
0810 18:03:57.373095 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:57.373146 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 18:03:57.373853 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:57.373670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:34369)
0810 18:03:57.373763 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:57.373804 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 18:03:57.374048 12266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:03:57.377686 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 12133
I20260810 18:03:58.047538 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:58.047120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:50471)
0810 18:03:58.047367 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:58.047421 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260810 18:03:58.191429 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:58.190981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:36071)
0810 18:03:58.191280 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:58.191335 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260810 18:03:58.374980 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:58.374574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:36309)
0810 18:03:58.374829 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:58.374885 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":70,"threads_started":1}
I20260810 18:03:58.485045 12273 raft_consensus.cc:482] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:03:58.485275 12273 raft_consensus.cc:504] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:03:58.486353 12273 leader_election.cc:288] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333), 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625)
Aug 10 18:03:58 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.67@KRBTEST.COM
Aug 10 18:03:58 dist-test-slave-srgv krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.66@KRBTEST.COM
I20260810 18:03:58.501607 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:58.486585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:46625 (local address 127.28.83.65:59509)
0810 18:03:58.486870 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:58.486887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:58.487018 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:58.488828 (+ 1810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:58.488834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:58.489469 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:58.489985 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:58.490009 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:58.491350 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:58.491356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:58.492357 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:58.492366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:58.493111 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:58.493114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:58.493302 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:58.494107 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:58.494163 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:58.495818 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:58.499282 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:58.499284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:58.499285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:58.499428 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:58.499758 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:58.499763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:58.499768 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:58.499851 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:58.500758 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:58.500761 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:58.500833 (+ 72us) client_negotiation.cc:769] Sending connection context
0810 18:03:58.500980 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 18:03:58.501393 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":195,"spinlock_wait_cycles":11264,"thread_start_us":65,"threads_started":1}
I20260810 18:03:58.501637 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:58.486782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:59509 (local address 127.28.83.67:46625)
0810 18:03:58.487178 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:58.487217 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:58.487254 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:03:58.487915 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:58.487920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:58.488266 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:58.488791 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:58.490211 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:58.491181 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:58.492485 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:58.492987 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:58.493168 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:58.496427 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:58.496432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:58.496442 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:58.496474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:58.498798 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:58.499534 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:58.499540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:58.499545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:58.499662 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:58.499940 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:58.499943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:58.499945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:58.500365 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:58.500508 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:58.501128 (+ 620us) server_negotiation.cc:299] Negotiation successful
0810 18:03:58.501449 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
I20260810 18:03:58.501607 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:58.486575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41333 (local address 127.28.83.65:35761)
0810 18:03:58.486762 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:58.486778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:03:58.486885 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:03:58.488833 (+ 1948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:03:58.488837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:03:58.489465 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:03:58.489998 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:58.490009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:58.491459 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:58.491463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:58.492352 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:03:58.492362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:03:58.493154 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:03:58.493158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:03:58.493303 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:58.494074 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:03:58.494090 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:03:58.496198 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:03:58.498933 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:58.498937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:58.498948 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:58.499195 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:58.499549 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:03:58.499551 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:03:58.499552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:03:58.499683 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:03:58.500496 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:03:58.500502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:03:58.500772 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 18:03:58.500980 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 18:03:58.501473 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":312,"spinlock_wait_cycles":96640}
I20260810 18:03:58.501637 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:58.486899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:35761 (local address 127.28.83.66:41333)
0810 18:03:58.487275 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0810 18:03:58.487295 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:03:58.487329 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:03:58.487949 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:03:58.487954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:03:58.488262 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:03:58.488792 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:03:58.490183 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:58.491283 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:58.492483 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:03:58.493005 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:03:58.493201 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:03:58.496419 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:03:58.496425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:03:58.496441 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:03:58.496474 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:03:58.498811 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:58.499303 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:58.499307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:58.499311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:58.499424 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:03:58.499810 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:03:58.499813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:03:58.499814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:03:58.500365 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:03:58.500507 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 18:03:58.501126 (+ 619us) server_negotiation.cc:299] Negotiation successful
0810 18:03:58.501456 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":104,"threads_started":1}
W20260810 18:03:58.504202 11871 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
W20260810 18:03:58.504400 11873 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
I20260810 18:03:58.504518 11873 leader_election.cc:302] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffe36616bec44765a949d3a96e04952a; no voters: 8451c71ffbea48ce91d5efd293c37643, fe73cd93bb5d4cae839fb9f590709106
I20260810 18:03:58.504802 12273 raft_consensus.cc:2726] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:03:59.048787 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:59.048366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:35987)
0810 18:03:59.048637 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:59.048689 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260810 18:03:59.192474 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:59.192101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:33889)
0810 18:03:59.192339 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:59.192389 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260810 18:03:59.376222 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:03:59.375811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:58079)
0810 18:03:59.376053 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 18:03:59.376126 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260810 18:04:00.050056 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:00.049617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:48933)
0810 18:04:00.049892 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:00.049954 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":101,"threads_started":1}
I20260810 18:04:00.111320 12281 raft_consensus.cc:482] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:04:00.111510 12281 raft_consensus.cc:504] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:04:00.112094 12281 leader_election.cc:288] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333), 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625)
W20260810 18:04:00.113073 11871 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
W20260810 18:04:00.113252 11873 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
I20260810 18:04:00.113318 11873 leader_election.cc:302] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffe36616bec44765a949d3a96e04952a; no voters: 8451c71ffbea48ce91d5efd293c37643, fe73cd93bb5d4cae839fb9f590709106
I20260810 18:04:00.113524 12281 raft_consensus.cc:2726] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:04:00.193820 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:00.193337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:60369)
0810 18:04:00.193657 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:00.193724 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":132,"threads_started":1}
I20260810 18:04:00.377559 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:00.376999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:47963)
0810 18:04:00.377320 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:00.377392 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260810 18:04:01.051155 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:01.050735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:47971)
0810 18:04:01.050996 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:01.051051 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260810 18:04:01.194958 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:01.194534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:46289)
0810 18:04:01.194792 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:01.194849 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260810 18:04:01.378814 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:01.378361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:47997)
0810 18:04:01.378647 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:01.378720 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":102,"threads_started":1}
I20260810 18:04:02.052667 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:02.052206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:42775)
0810 18:04:02.052508 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:02.052569 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":79,"threads_started":1}
I20260810 18:04:02.196100 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:02.195682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:51685)
0810 18:04:02.195956 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:02.196009 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":107,"threads_started":1}
I20260810 18:04:02.379940 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:02.379541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:54197)
0810 18:04:02.379796 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:02.379850 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
I20260810 18:04:02.559566 12290 raft_consensus.cc:482] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:04:02.559712 12290 raft_consensus.cc:504] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:04:02.560168 12290 leader_election.cc:288] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333), 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625)
W20260810 18:04:02.560894 11873 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
W20260810 18:04:02.560978 11871 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
I20260810 18:04:02.561077 11871 leader_election.cc:302] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffe36616bec44765a949d3a96e04952a; no voters: 8451c71ffbea48ce91d5efd293c37643, fe73cd93bb5d4cae839fb9f590709106
I20260810 18:04:02.561228 12290 raft_consensus.cc:2726] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:04:03.053996 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:03.053595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:36153)
0810 18:04:03.053842 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:03.053901 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260810 18:04:03.197160 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:03.196755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:60943)
0810 18:04:03.197017 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:03.197066 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
I20260810 18:04:03.381101 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:03.380654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:53317)
0810 18:04:03.380933 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:03.380994 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260810 18:04:04.055299 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:04.054838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:48985)
0810 18:04:04.055127 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:04.055191 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260810 18:04:04.198380 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:04.197904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:40139)
0810 18:04:04.198218 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:04.198274 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":99,"threads_started":1}
I20260810 18:04:04.382391 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:04.381891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:52269)
0810 18:04:04.382239 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:04.382297 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":88,"threads_started":1}
I20260810 18:04:04.629709 12297 raft_consensus.cc:482] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:04:04.629841 12297 raft_consensus.cc:504] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:04:04.630190 12297 leader_election.cc:288] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333), 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625)
W20260810 18:04:04.630987 11873 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
W20260810 18:04:04.631093 11871 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
I20260810 18:04:04.631152 11871 leader_election.cc:302] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffe36616bec44765a949d3a96e04952a; no voters: 8451c71ffbea48ce91d5efd293c37643, fe73cd93bb5d4cae839fb9f590709106
I20260810 18:04:04.631295 12297 raft_consensus.cc:2726] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:04:05.056685 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:05.056079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:45335)
0810 18:04:05.056464 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:05.056537 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":149,"threads_started":1}
I20260810 18:04:05.199570 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:05.199163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:59365)
0810 18:04:05.199423 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:05.199478 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20260810 18:04:05.383550 12300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:05.383152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:46531)
0810 18:04:05.383405 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:05.383462 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260810 18:04:06.057786 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:06.057365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:53791)
0810 18:04:06.057638 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:06.057690 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":121,"threads_started":1}
I20260810 18:04:06.200644 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:06.200234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:60839)
0810 18:04:06.200500 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:06.200554 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260810 18:04:06.384724 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:06.384263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:50141)
0810 18:04:06.384564 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:06.384631 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260810 18:04:07.059088 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:07.058619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:40795)
0810 18:04:07.058902 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:07.058955 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260810 18:04:07.201830 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:07.201391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:38513)
0810 18:04:07.201688 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:07.201741 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":73,"threads_started":1}
I20260810 18:04:07.386001 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:07.385589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:56011)
0810 18:04:07.385855 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:07.385908 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260810 18:04:08.060223 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:08.059788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:55179)
0810 18:04:08.060072 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:08.060122 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":78,"threads_started":1}
I20260810 18:04:08.202903 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:08.202515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:35865)
0810 18:04:08.202756 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:08.202810 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260810 18:04:08.387140 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:08.386745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:40305)
0810 18:04:08.386981 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:08.387038 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260810 18:04:09.061326 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:09.060918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:50109)
0810 18:04:09.061163 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:09.061228 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260810 18:04:09.204026 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:09.203632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:37865)
0810 18:04:09.203872 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:09.203931 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260810 18:04:09.388500 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:09.387883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:43519)
0810 18:04:09.388303 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:09.388381 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":104,"threads_started":1}
I20260810 18:04:10.062649 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:10.062219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:40925)
0810 18:04:10.062497 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:10.062553 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260810 18:04:10.205130 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:10.204741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:48673)
0810 18:04:10.204980 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:10.205031 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260810 18:04:10.389778 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:10.389309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:57929)
0810 18:04:10.389630 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:10.389685 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":129,"threads_started":1}
I20260810 18:04:11.063979 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:11.063468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:54553)
0810 18:04:11.063786 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:11.063859 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
I20260810 18:04:11.161980 12317 raft_consensus.cc:482] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:04:11.162114 12317 raft_consensus.cc:504] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe73cd93bb5d4cae839fb9f590709106" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41333 } } peers { permanent_uuid: "8451c71ffbea48ce91d5efd293c37643" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 46625 } } peers { permanent_uuid: "ffe36616bec44765a949d3a96e04952a" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 36635 } }
I20260810 18:04:11.162586 12317 leader_election.cc:288] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333), 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625)
W20260810 18:04:11.163249 11873 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe73cd93bb5d4cae839fb9f590709106 (127.28.83.66:41333): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
W20260810 18:04:11.163357 11871 leader_election.cc:341] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8451c71ffbea48ce91d5efd293c37643 (127.28.83.67:46625): Not found: Tablet not found: 1b1082e3e4cf455e92f5987fe0caf3e2
I20260810 18:04:11.163475 11871 leader_election.cc:302] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffe36616bec44765a949d3a96e04952a; no voters: 8451c71ffbea48ce91d5efd293c37643, fe73cd93bb5d4cae839fb9f590709106
I20260810 18:04:11.163599 12317 raft_consensus.cc:2726] T 1b1082e3e4cf455e92f5987fe0caf3e2 P ffe36616bec44765a949d3a96e04952a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:04:11.206281 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:11.205856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:46057)
0810 18:04:11.206113 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:11.206168 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260810 18:04:11.390954 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:11.390553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:40621)
0810 18:04:11.390807 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:11.390861 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20260810 18:04:12.065639 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:12.065088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.65:58551)
0810 18:04:12.065453 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:12.065530 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
I20260810 18:04:12.207674 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:12.207244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.66:45259)
0810 18:04:12.207506 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:12.207567 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260810 18:04:12.381281 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 11817
I20260810 18:04:12.388485 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 11994
I20260810 18:04:12.392264 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:12.391880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:36225 (local address 127.28.83.67:60035)
0810 18:04:12.392118 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:12.392167 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260810 18:04:12.394433 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 12133
2026-08-10T18:04:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19040 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:04:12 dist-test-slave-srgv krb5kdc[12325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:04:12 dist-test-slave-srgv krb5kdc[12325](info): set up 2 sockets
Aug 10 18:04:12 dist-test-slave-srgv krb5kdc[12325](info): commencing operation
krb5kdc: starting...
W20260810 18:04:13.456532 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:04:13 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:04:13.576838 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37985
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:04:13.684830 12338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:13.685120 12338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:13.685191 12338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:13.689030 12338 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 18:04:13.689103 12338 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 18:04:13.689131 12338 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 18:04:13.689160 12338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:04:13.689203 12338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:13.689241 12338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:13.689272 12338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:04:13.689304 12338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:04:13.694300 12338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:37985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:13.695302 12338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:13.696312 12338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:13.704864 12346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:13.704895 12344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:13.704864 12343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:13.704931 12338 server_base.cc:1034] running on GCE node
I20260810 18:04:13.705806 12338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:13.706862 12338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:13.708045 12338 hybrid_clock.cc:648] HybridClock initialized: now 1786385053708026 us; error 30 us; skew 500 ppm
Aug 10 18:04:13 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385053, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:13.710781 12338 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:04:13.711617 12338 webserver.cc:460] Webserver started at http://127.28.83.126:39579/ using document root <none> and password file <none>
I20260810 18:04:13.712117 12338 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:13.712172 12338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:13.712337 12338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:04:13.714337 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "2611d4f76bf44d3c91d0173723804cf3"
format_stamp: "Formatted at 2026-08-10 18:04:13 on dist-test-slave-srgv"
server_key: "0f7f435462a1fa52fff193455598aac5"
server_key_iv: "5eb25f5a22dd0f86cd41fdcb1b822e58"
server_key_version: "encryptionkey@0"
I20260810 18:04:13.714864 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "2611d4f76bf44d3c91d0173723804cf3"
format_stamp: "Formatted at 2026-08-10 18:04:13 on dist-test-slave-srgv"
server_key: "0f7f435462a1fa52fff193455598aac5"
server_key_iv: "5eb25f5a22dd0f86cd41fdcb1b822e58"
server_key_version: "encryptionkey@0"
I20260810 18:04:13.718341 12338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:04:13.720557 12354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:13.721814 12338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:04:13.721911 12338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "2611d4f76bf44d3c91d0173723804cf3"
format_stamp: "Formatted at 2026-08-10 18:04:13 on dist-test-slave-srgv"
server_key: "0f7f435462a1fa52fff193455598aac5"
server_key_iv: "5eb25f5a22dd0f86cd41fdcb1b822e58"
server_key_version: "encryptionkey@0"
I20260810 18:04:13.721992 12338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:13.742403 12338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:13.745718 12338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:13.745877 12338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:13.752600 12338 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:37985
I20260810 18:04:13.752880 12409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:37985 every 8 connection(s)
I20260810 18:04:13.753600 12338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:04:13.756313 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:04:13.762041 12410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Bootstrap starting.
I20260810 18:04:13.762851 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 12338
I20260810 18:04:13.762948 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:04:13.763239 29005 external_mini_cluster.cc:1428] Setting key 2555697e488bd078d5dbb96f7fb280ef
I20260810 18:04:13.764838 12410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Neither blocks nor log segments found. Creating new log.
I20260810 18:04:13.765652 12410 log.cc:824] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Log is configured to *not* fsync() on all Append() calls
I20260810 18:04:13.767786 12410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: No bootstrap required, opened a new log
I20260810 18:04:13.771876 12410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:13.772145 12410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:04:13.772218 12410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2611d4f76bf44d3c91d0173723804cf3, State: Initialized, Role: FOLLOWER
Aug 10 18:04:13 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:04:13.772650 12410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:13.772804 12410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:04:13.772888 12410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:04:13.772993 12410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:04:13.774314 12410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:13.774672 12410 leader_election.cc:302] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2611d4f76bf44d3c91d0173723804cf3; no voters:
I20260810 18:04:13.775031 12410 leader_election.cc:288] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:04:13.775172 12415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:04:13.775452 12415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 2611d4f76bf44d3c91d0173723804cf3, State: Running, Role: LEADER
I20260810 18:04:13.775838 12415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:13.776350 12410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:04:13.777558 12417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2611d4f76bf44d3c91d0173723804cf3. Latest consensus state: current_term: 1 leader_uuid: "2611d4f76bf44d3c91d0173723804cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } } }
I20260810 18:04:13.777697 12417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:04:13.778105 12425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:04:13.778065 12416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2611d4f76bf44d3c91d0173723804cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } } }
I20260810 18:04:13.778294 12416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:04:13.779054 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:13.764431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.28.83.126:37985)
0810 18:04:13.764884 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:13.764899 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:13.764933 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:04:13.765672 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:13.765689 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:13.766060 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:13.766325 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:13.767059 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:13.768483 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:13.769321 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:13.770097 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:13.770288 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:13.772991 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:13.772999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:13.773036 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:13.773068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:13.775201 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:13.775794 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:13.775800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:13.775805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:13.775942 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:13.776257 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:13.776261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:13.776262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:13.776802 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:13.776977 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:13.777814 (+ 837us) server_negotiation.cc:299] Negotiation successful
0810 18:04:13.778137 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":152,"threads_started":1}
I20260810 18:04:13.782374 12425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:04:13.788089 12425 catalog_manager.cc:1347] Generated new cluster ID: 6e25583b5fe74c598fe461c4646150fa
I20260810 18:04:13.788156 12425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:04:13.800026 12425 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:04:13.800918 12425 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:04:13.811825 12425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Generated new TSK 0
I20260810 18:04:13.812556 12425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:04:13.846637 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:37985
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:04:13.995420 12437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:13.995662 12437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:13.995743 12437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:14.000001 12437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:14.000067 12437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:14.000191 12437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:04:14.005277 12437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:14.006271 12437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:14.007249 12437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:14.016597 12445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:14.016587 12443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:14.016662 12437 server_base.cc:1034] running on GCE node
W20260810 18:04:14.016569 12442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:14.017264 12437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:14.017961 12437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:14.019147 12437 hybrid_clock.cc:648] HybridClock initialized: now 1786385054019132 us; error 30 us; skew 500 ppm
Aug 10 18:04:14 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385054, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:14.021898 12437 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:04:14.022831 12437 webserver.cc:460] Webserver started at http://127.28.83.65:34585/ using document root <none> and password file <none>
I20260810 18:04:14.023411 12437 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:14.023499 12437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:14.023698 12437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:04:14.026117 12437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "c68dfee61eea40509b6e6c38912b4837"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "5c0488531102411f2bc45a9a0c857f9a"
server_key_iv: "e2bd9361c9b66eb8168d3fd45bd4981f"
server_key_version: "encryptionkey@0"
I20260810 18:04:14.026674 12437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "c68dfee61eea40509b6e6c38912b4837"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "5c0488531102411f2bc45a9a0c857f9a"
server_key_iv: "e2bd9361c9b66eb8168d3fd45bd4981f"
server_key_version: "encryptionkey@0"
I20260810 18:04:14.030125 12437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:04:14.032361 12452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:14.033660 12437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:04:14.033803 12437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "c68dfee61eea40509b6e6c38912b4837"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "5c0488531102411f2bc45a9a0c857f9a"
server_key_iv: "e2bd9361c9b66eb8168d3fd45bd4981f"
server_key_version: "encryptionkey@0"
I20260810 18:04:14.033919 12437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:14.049556 12437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:14.052438 12437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:14.052651 12437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:14.053264 12437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:04:14.054195 12437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:04:14.054267 12437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:14.054343 12437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:04:14.054387 12437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:14.063824 12437 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:43303
I20260810 18:04:14.063855 12569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:43303 every 8 connection(s)
I20260810 18:04:14.064743 12437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:04:14.074004 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 12437
I20260810 18:04:14.074152 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:04:14.074440 29005 external_mini_cluster.cc:1428] Setting key 762ea2793b286b3501ee70b026af55b0
Aug 10 18:04:14 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385054, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:04:14.080371 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.066549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:49833 (local address 127.28.83.126:37985)
0810 18:04:14.066694 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.066698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.067494 (+ 796us) server_negotiation.cc:407] Connection header received
0810 18:04:14.068855 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.068858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.068926 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.069032 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:14.070342 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.070902 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.072019 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.072272 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.072407 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.075228 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:14.075233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:14.075239 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:14.075271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:14.078020 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:14.078622 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:14.078627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:14.078629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:14.078732 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:14.079072 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:14.079077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:14.079079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:14.079336 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:14.079451 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.079935 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.080115 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:04:14.081161 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.066920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:49833)
0810 18:04:14.067342 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:14.067384 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:14.068635 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:14.069203 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:14.069209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:14.069701 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:04:14.070183 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.070197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.071031 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.071035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.071891 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.071900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.072464 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.072468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.072661 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.073304 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:04:14.073321 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:04:14.075079 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:04:14.078203 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:14.078220 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:14.078241 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:14.078520 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:14.078831 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:14.078833 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:14.078835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:14.078972 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:14.079445 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:04:14.079450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:04:14.079686 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 18:04:14.079850 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 18:04:14.080114 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
I20260810 18:04:14.082525 12570 heartbeater.cc:347] Connected to a master server at 127.28.83.126:37985
I20260810 18:04:14.082815 12570 heartbeater.cc:464] Registering TS with master...
I20260810 18:04:14.083429 12570 heartbeater.cc:511] Master 127.28.83.126:37985 requested a full tablet report, sending...
I20260810 18:04:14.085008 12372 ts_manager.cc:194] Registered new tserver with Master: c68dfee61eea40509b6e6c38912b4837 (127.28.83.65:43303)
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:04:14.086191 12372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:49833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:04:14.102334 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:37985
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:04:14.212538 12576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:14.212865 12576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:14.212945 12576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:14.217056 12576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:14.217142 12576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:14.217232 12576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:04:14.222666 12576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:14.223820 12576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:14.224874 12576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:14.234840 12581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:14.234848 12582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:14.234790 12584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:14.234980 12576 server_base.cc:1034] running on GCE node
I20260810 18:04:14.235523 12576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:14.236167 12576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:14.237329 12576 hybrid_clock.cc:648] HybridClock initialized: now 1786385054237297 us; error 44 us; skew 500 ppm
Aug 10 18:04:14 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385054, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:14.240173 12576 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:04:14.241043 12576 webserver.cc:460] Webserver started at http://127.28.83.66:46067/ using document root <none> and password file <none>
I20260810 18:04:14.241665 12576 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:14.241762 12576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:14.242046 12576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:04:14.244055 12576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "c2d696c7201e4d84b764659b93456c2e"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "6a60bee3e735320fe919413edb3df9fe"
server_key_iv: "15ea84207b6faee1f5a5e10accbe3b2d"
server_key_version: "encryptionkey@0"
I20260810 18:04:14.244628 12576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "c2d696c7201e4d84b764659b93456c2e"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "6a60bee3e735320fe919413edb3df9fe"
server_key_iv: "15ea84207b6faee1f5a5e10accbe3b2d"
server_key_version: "encryptionkey@0"
I20260810 18:04:14.248195 12576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 18:04:14.250535 12593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:14.251734 12576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:04:14.251864 12576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "c2d696c7201e4d84b764659b93456c2e"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "6a60bee3e735320fe919413edb3df9fe"
server_key_iv: "15ea84207b6faee1f5a5e10accbe3b2d"
server_key_version: "encryptionkey@0"
I20260810 18:04:14.251973 12576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:14.274026 12576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:14.277769 12576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:14.278029 12576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:14.278647 12576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:04:14.279429 12576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:04:14.279476 12576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:14.279515 12576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:04:14.279536 12576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:14.288777 12576 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:40429
I20260810 18:04:14.288802 12707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:40429 every 8 connection(s)
I20260810 18:04:14.289764 12576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
Aug 10 18:04:14 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385054, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:04:14.299372 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 12576
I20260810 18:04:14.299497 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:04:14.299852 29005 external_mini_cluster.cc:1428] Setting key 404a94c9cd1f1825c3336b14f117d3d4
I20260810 18:04:14.303407 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.291644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:42345 (local address 127.28.83.126:37985)
0810 18:04:14.291782 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.291786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.292614 (+ 828us) server_negotiation.cc:407] Connection header received
0810 18:04:14.293896 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.293899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.293945 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.294019 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:14.295116 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.295670 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.296743 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.296961 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.297036 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.299458 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:14.299463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:14.299467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:14.299497 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:14.301183 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:14.301826 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:14.301830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:14.301832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:14.301921 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:14.302299 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:14.302302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:14.302304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:14.302492 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:14.302599 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.303106 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.303242 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:04:14.304241 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.292025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:42345)
0810 18:04:14.292462 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:14.292504 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:14.293695 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:14.294174 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:14.294180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:14.294525 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:04:14.294991 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.295004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.295781 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.295784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.296646 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.296654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.297104 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.297107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.297245 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.297842 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:04:14.297857 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:04:14.299289 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:04:14.301313 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:14.301318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:14.301357 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:14.301718 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:14.302027 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:14.302032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:14.302034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:14.302191 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:14.302599 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:04:14.302605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:04:14.302860 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 18:04:14.303049 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 18:04:14.303383 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260810 18:04:14.305493 12708 heartbeater.cc:347] Connected to a master server at 127.28.83.126:37985
I20260810 18:04:14.305739 12708 heartbeater.cc:464] Registering TS with master...
I20260810 18:04:14.306284 12708 heartbeater.cc:511] Master 127.28.83.126:37985 requested a full tablet report, sending...
I20260810 18:04:14.307365 12372 ts_manager.cc:194] Registered new tserver with Master: c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429)
I20260810 18:04:14.307940 12372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:42345
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:04:14.328600 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:37985
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:04:14.440275 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:14.440564 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:14.440649 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:14.444721 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:14.444818 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:14.444914 12715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:04:14.450174 12715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:14.451248 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:14.452244 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:14.462157 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:14.462244 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:14.462258 12715 server_base.cc:1034] running on GCE node
W20260810 18:04:14.462157 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:14.462874 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:14.463460 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:14.464628 12715 hybrid_clock.cc:648] HybridClock initialized: now 1786385054464601 us; error 48 us; skew 500 ppm
Aug 10 18:04:14 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385054, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:14.467496 12715 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:04:14.468389 12715 webserver.cc:460] Webserver started at http://127.28.83.67:37095/ using document root <none> and password file <none>
I20260810 18:04:14.468932 12715 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:14.468994 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:14.469226 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:04:14.471241 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "a58e77dd57524053b593d3db189d76a1"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "2fe63b8c9c0e6e57386cfce569b87ef0"
server_key_iv: "c67354a4aaccf29d2d52b3da5788e69a"
server_key_version: "encryptionkey@0"
I20260810 18:04:14.471804 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "a58e77dd57524053b593d3db189d76a1"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "2fe63b8c9c0e6e57386cfce569b87ef0"
server_key_iv: "c67354a4aaccf29d2d52b3da5788e69a"
server_key_version: "encryptionkey@0"
I20260810 18:04:14.475533 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:04:14.478161 12731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:14.479447 12715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:04:14.479606 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "a58e77dd57524053b593d3db189d76a1"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "2fe63b8c9c0e6e57386cfce569b87ef0"
server_key_iv: "c67354a4aaccf29d2d52b3da5788e69a"
server_key_version: "encryptionkey@0"
I20260810 18:04:14.479723 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:14.504601 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:14.507683 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:14.507894 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:14.508558 12715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:04:14.509389 12715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:04:14.509496 12715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:14.509564 12715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:04:14.509604 12715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:14.519017 12715 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:40221
I20260810 18:04:14.519042 12845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:40221 every 8 connection(s)
I20260810 18:04:14.519970 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:04:14.525456 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 12715
I20260810 18:04:14.525592 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:04:14.525909 29005 external_mini_cluster.cc:1428] Setting key 05cc11a6b624447d1246d6cf439254da
Aug 10 18:04:14 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385054, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:04:14.534329 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.523175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:51513 (local address 127.28.83.126:37985)
0810 18:04:14.523326 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.523330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.523345 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:04:14.524189 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.524192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.524235 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.524312 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:14.525732 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.526413 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.528065 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.528309 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.528389 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.530894 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:14.530897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:14.530899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:14.530924 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:14.532440 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:14.532924 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:14.532927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:14.532928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:14.533003 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:14.533309 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:14.533311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:14.533312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:14.533558 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:14.533638 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.534107 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.534207 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:04:14.535154 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.522242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:51513)
0810 18:04:14.522700 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:14.522744 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:14.523989 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:14.524473 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:14.524479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:14.524870 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:04:14.525584 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.525607 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.526564 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.526566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.527899 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.527924 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.528438 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.528441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.528580 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.529149 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:04:14.529165 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:04:14.530742 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:04:14.532563 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:14.532568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:14.532580 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:14.532834 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:14.533084 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:14.533086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:14.533087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:14.533225 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:14.533650 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:04:14.533655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:04:14.533890 (+ 235us) client_negotiation.cc:769] Sending connection context
0810 18:04:14.534049 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 18:04:14.534314 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
I20260810 18:04:14.536326 12846 heartbeater.cc:347] Connected to a master server at 127.28.83.126:37985
I20260810 18:04:14.536566 12846 heartbeater.cc:464] Registering TS with master...
I20260810 18:04:14.537050 12846 heartbeater.cc:511] Master 127.28.83.126:37985 requested a full tablet report, sending...
I20260810 18:04:14.538151 12372 ts_manager.cc:194] Registered new tserver with Master: a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221)
I20260810 18:04:14.538795 12372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:51513
I20260810 18:04:14.539951 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:04:14.548811 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.542019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.28.83.126:37985)
0810 18:04:14.542136 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.542139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.542339 (+ 200us) server_negotiation.cc:407] Connection header received
0810 18:04:14.542426 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.542428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.542462 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.542526 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:14.543218 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.543693 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.544476 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.544671 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.544751 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.546068 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:14.546071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:14.546072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:14.546093 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:14.547511 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:14.547872 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:14.547874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:14.547875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:14.547944 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:14.548156 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:14.548158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:14.548159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:14.548348 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:14.548445 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.548609 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.548693 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:04:14.551667 12372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:04:14.554770 12372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:04:14.568626 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.563047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.28.83.67:40221)
0810 18:04:14.563576 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.563589 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.563616 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:04:14.563922 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.563928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.564164 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.564327 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:04:14.564813 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.565924 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.567334 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.567912 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.568036 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.568199 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.568355 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.568467 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":165,"threads_started":1}
I20260810 18:04:14.568614 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.562879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:40221 (local address 127.0.0.1:37116)
0810 18:04:14.563263 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:14.563315 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:14.563450 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:14.564528 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:14.564536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:14.564558 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:04:14.564692 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.564696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.566053 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.566056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.567214 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.567223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.568037 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.568040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.568155 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.568193 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 18:04:14.568326 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 18:04:14.568465 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":101,"threads_started":1}
I20260810 18:04:14.569487 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.563346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:43303 (local address 127.0.0.1:48746)
0810 18:04:14.564369 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:14.564381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:14.564502 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:14.565002 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:14.565005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:14.565023 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:04:14.565114 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.565120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.566644 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.566648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.568013 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.568023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.569079 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.569083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.569169 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.569191 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:04:14.569267 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:04:14.569330 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":23,"thread_start_us":97,"threads_started":1}
I20260810 18:04:14.569577 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.563782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.28.83.65:43303)
0810 18:04:14.564217 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.564235 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.564478 (+ 243us) server_negotiation.cc:407] Connection header received
0810 18:04:14.564647 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.564653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.564852 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.565003 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:04:14.565212 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.566469 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.568143 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.568952 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.569066 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.569200 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.569294 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.569389 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":107,"threads_started":1}
I20260810 18:04:14.571669 12780 tablet_service.cc:1467] Processing CreateTablet for tablet b820ff37949e4886be1119a1ae1a7d9d (DEFAULT_TABLE table=test-table [id=79e6d91faca84299a5a409c4b94cb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:04:14.571805 12502 tablet_service.cc:1467] Processing CreateTablet for tablet b820ff37949e4886be1119a1ae1a7d9d (DEFAULT_TABLE table=test-table [id=79e6d91faca84299a5a409c4b94cb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:04:14.572590 12780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b820ff37949e4886be1119a1ae1a7d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:04:14.572724 12502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b820ff37949e4886be1119a1ae1a7d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:04:14.574227 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.563606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:40429 (local address 127.0.0.1:51890)
0810 18:04:14.564366 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:14.564382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:14.564461 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:14.565248 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:14.565251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:14.565265 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:04:14.565350 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.565354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.566712 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.566715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.567901 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.567911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.573775 (+ 5864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.573778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.573907 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.573932 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:04:14.574013 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 18:04:14.574091 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260810 18:04:14.574747 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.563933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.28.83.66:40429)
0810 18:04:14.564780 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.564791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.564813 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:04:14.564883 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.564888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.565056 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.565191 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:04:14.565471 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.566577 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.568042 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.569267 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.569367 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.569514 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.574482 (+ 4968us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.574595 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":142,"threads_started":1}
I20260810 18:04:14.577661 12642 tablet_service.cc:1467] Processing CreateTablet for tablet b820ff37949e4886be1119a1ae1a7d9d (DEFAULT_TABLE table=test-table [id=79e6d91faca84299a5a409c4b94cb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:04:14.578805 12642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b820ff37949e4886be1119a1ae1a7d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:04:14.579156 12867 tablet_bootstrap.cc:492] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Bootstrap starting.
I20260810 18:04:14.580272 12868 tablet_bootstrap.cc:492] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: Bootstrap starting.
I20260810 18:04:14.581854 12867 tablet_bootstrap.cc:654] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Neither blocks nor log segments found. Creating new log.
I20260810 18:04:14.582633 12867 log.cc:824] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Log is configured to *not* fsync() on all Append() calls
I20260810 18:04:14.582824 12868 tablet_bootstrap.cc:654] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: Neither blocks nor log segments found. Creating new log.
I20260810 18:04:14.583664 12868 log.cc:824] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: Log is configured to *not* fsync() on all Append() calls
I20260810 18:04:14.584775 12867 tablet_bootstrap.cc:492] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: No bootstrap required, opened a new log
I20260810 18:04:14.584975 12867 ts_tablet_manager.cc:1397] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:04:14.585695 12868 tablet_bootstrap.cc:492] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: No bootstrap required, opened a new log
I20260810 18:04:14.585877 12868 ts_tablet_manager.cc:1397] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:04:14.587337 12871 tablet_bootstrap.cc:492] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: Bootstrap starting.
I20260810 18:04:14.589232 12867 raft_consensus.cc:348] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:14.589533 12867 raft_consensus.cc:374] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:04:14.589602 12867 raft_consensus.cc:729] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68dfee61eea40509b6e6c38912b4837, State: Initialized, Role: FOLLOWER
I20260810 18:04:14.589687 12871 tablet_bootstrap.cc:654] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: Neither blocks nor log segments found. Creating new log.
I20260810 18:04:14.589525 12868 raft_consensus.cc:348] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:14.589785 12868 raft_consensus.cc:374] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:04:14.589854 12868 raft_consensus.cc:729] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58e77dd57524053b593d3db189d76a1, State: Initialized, Role: FOLLOWER
I20260810 18:04:14.590111 12867 consensus_queue.cc:260] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:14.590255 12868 consensus_queue.cc:260] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:14.590557 12871 log.cc:824] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:04:14.591025 12846 heartbeater.cc:503] Master 127.28.83.126:37985 was elected leader, sending a full tablet report...
I20260810 18:04:14.591151 12868 ts_tablet_manager.cc:1428] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:04:14.591151 12867 ts_tablet_manager.cc:1428] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:04:14.592475 12871 tablet_bootstrap.cc:492] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: No bootstrap required, opened a new log
I20260810 18:04:14.592686 12871 ts_tablet_manager.cc:1397] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:04:14.593600 12570 heartbeater.cc:503] Master 127.28.83.126:37985 was elected leader, sending a full tablet report...
I20260810 18:04:14.596174 12871 raft_consensus.cc:348] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:14.596401 12871 raft_consensus.cc:374] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:04:14.596459 12871 raft_consensus.cc:729] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d696c7201e4d84b764659b93456c2e, State: Initialized, Role: FOLLOWER
I20260810 18:04:14.596827 12871 consensus_queue.cc:260] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:14.597506 12708 heartbeater.cc:503] Master 127.28.83.126:37985 was elected leader, sending a full tablet report...
I20260810 18:04:14.597759 12871 ts_tablet_manager.cc:1428] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:04:14.697528 12875 raft_consensus.cc:482] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:04:14.697814 12875 raft_consensus.cc:504] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:14.699035 12875 leader_election.cc:288] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221), c68dfee61eea40509b6e6c38912b4837 (127.28.83.65:43303)
I20260810 18:04:14.704442 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.699523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:42323 (local address 127.28.83.65:43303)
0810 18:04:14.699687 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.699694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.699876 (+ 182us) server_negotiation.cc:407] Connection header received
0810 18:04:14.699939 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.699942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.699991 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.700073 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:04:14.700342 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.701084 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.703052 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.703648 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.703777 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.703832 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.704229 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.704302 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:04:14.704442 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.699418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:47953 (local address 127.28.83.67:40221)
0810 18:04:14.699557 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.699563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.699578 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:04:14.699785 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.699788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.699856 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.699948 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:04:14.700270 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.701084 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.703052 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.703644 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.703778 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.703832 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.704224 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.704302 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:04:14.704499 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.699269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:40221 (local address 127.28.83.66:47953)
0810 18:04:14.699486 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:14.699503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:14.699619 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:14.699982 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:14.699985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:14.699997 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:04:14.700145 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.700151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.701206 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.701209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.702901 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.702914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.703815 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.703820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.704028 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.704081 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 18:04:14.704192 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 18:04:14.704317 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":334}
I20260810 18:04:14.704507 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.699335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:43303 (local address 127.28.83.66:42323)
0810 18:04:14.699780 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:14.699791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:14.699867 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:14.700103 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:14.700107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:14.700122 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:04:14.700223 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.700231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.701206 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.701209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.702901 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:14.702914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.703815 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.703820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:14.704080 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.704103 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:04:14.704192 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 18:04:14.704316 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":226,"thread_start_us":142,"threads_started":1}
I20260810 18:04:14.705332 12800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b820ff37949e4886be1119a1ae1a7d9d" candidate_uuid: "c2d696c7201e4d84b764659b93456c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58e77dd57524053b593d3db189d76a1" is_pre_election: true
I20260810 18:04:14.705506 12523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b820ff37949e4886be1119a1ae1a7d9d" candidate_uuid: "c2d696c7201e4d84b764659b93456c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68dfee61eea40509b6e6c38912b4837" is_pre_election: true
I20260810 18:04:14.705757 12800 raft_consensus.cc:2454] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d696c7201e4d84b764659b93456c2e in term 0.
I20260810 18:04:14.705777 12523 raft_consensus.cc:2454] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d696c7201e4d84b764659b93456c2e in term 0.
I20260810 18:04:14.706363 12597 leader_election.cc:302] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a58e77dd57524053b593d3db189d76a1, c2d696c7201e4d84b764659b93456c2e, c68dfee61eea40509b6e6c38912b4837; no voters:
I20260810 18:04:14.706677 12875 raft_consensus.cc:2781] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:04:14.706782 12875 raft_consensus.cc:482] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:04:14.706844 12875 raft_consensus.cc:3037] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:04:14.707949 12875 raft_consensus.cc:504] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:14.708324 12875 leader_election.cc:288] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [CANDIDATE]: Term 1 election: Requested vote from peers a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221), c68dfee61eea40509b6e6c38912b4837 (127.28.83.65:43303)
I20260810 18:04:14.708732 12800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b820ff37949e4886be1119a1ae1a7d9d" candidate_uuid: "c2d696c7201e4d84b764659b93456c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58e77dd57524053b593d3db189d76a1"
I20260810 18:04:14.708884 12523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b820ff37949e4886be1119a1ae1a7d9d" candidate_uuid: "c2d696c7201e4d84b764659b93456c2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68dfee61eea40509b6e6c38912b4837"
I20260810 18:04:14.708884 12800 raft_consensus.cc:3037] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:04:14.709017 12523 raft_consensus.cc:3037] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:04:14.710316 12800 raft_consensus.cc:2454] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d696c7201e4d84b764659b93456c2e in term 1.
I20260810 18:04:14.710405 12523 raft_consensus.cc:2454] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d696c7201e4d84b764659b93456c2e in term 1.
I20260810 18:04:14.710686 12595 leader_election.cc:302] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a58e77dd57524053b593d3db189d76a1, c2d696c7201e4d84b764659b93456c2e; no voters:
I20260810 18:04:14.710891 12875 raft_consensus.cc:2781] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:04:14.711153 12875 raft_consensus.cc:686] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 1 LEADER]: Becoming Leader. State: Replica: c2d696c7201e4d84b764659b93456c2e, State: Running, Role: LEADER
I20260810 18:04:14.711503 12875 consensus_queue.cc:237] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:14.714823 12371 catalog_manager.cc:5495] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e reported cstate change: term changed from 0 to 1, leader changed from <none> to c2d696c7201e4d84b764659b93456c2e (127.28.83.66). New cstate: current_term: 1 leader_uuid: "c2d696c7201e4d84b764659b93456c2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } health_report { overall_health: UNKNOWN } } }
I20260810 18:04:14.759337 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.755546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.28.83.66:40429)
0810 18:04:14.755873 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:14.755877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:14.755887 (+ 10us) server_negotiation.cc:407] Connection header received
0810 18:04:14.755956 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:14.755958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:14.756010 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:14.756125 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:04:14.756405 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.756918 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.757857 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:14.758089 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:14.758198 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:14.758521 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:04:14.758911 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:04:14.759041 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:14.759107 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 18:04:14.759166 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260810 18:04:14.765077 12523 raft_consensus.cc:1261] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Refusing update from remote peer c2d696c7201e4d84b764659b93456c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:04:14.765079 12800 raft_consensus.cc:1261] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [term 1 FOLLOWER]: Refusing update from remote peer c2d696c7201e4d84b764659b93456c2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:04:14.765851 12875 consensus_queue.cc:1035] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:04:14.766042 12877 consensus_queue.cc:1035] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:04:14.770925 12847 tablet.cc:2369] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:04:14.774016 12882 mvcc.cc:205] Tried to move back new op lower bound from 7317033184310431744 to 7317033184101117952. Current Snapshot: MvccSnapshot[applied={T|T < 7317033184310431744}]
I20260810 18:04:14.776033 12880 mvcc.cc:205] Tried to move back new op lower bound from 7317033184310431744 to 7317033184101117952. Current Snapshot: MvccSnapshot[applied={T|T < 7317033184310431744}]
I20260810 18:04:14.783305 12371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:04:14.783507 12371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:04:14.786248 12371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Sending DeleteTablet for 3 replicas of tablet b820ff37949e4886be1119a1ae1a7d9d
I20260810 18:04:14.787256 12642 tablet_service.cc:1514] Processing DeleteTablet for tablet b820ff37949e4886be1119a1ae1a7d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:04:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:51890
I20260810 18:04:14.787250 12502 tablet_service.cc:1514] Processing DeleteTablet for tablet b820ff37949e4886be1119a1ae1a7d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:04:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:48746
I20260810 18:04:14.787609 12780 tablet_service.cc:1514] Processing DeleteTablet for tablet b820ff37949e4886be1119a1ae1a7d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:04:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:37116
I20260810 18:04:14.787791 12890 tablet_replica.cc:330] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: stopping tablet replica
I20260810 18:04:14.788122 12890 raft_consensus.cc:2229] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:04:14.788141 12892 tablet_replica.cc:330] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: stopping tablet replica
I20260810 18:04:14.788352 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 12437
I20260810 18:04:14.788491 12890 raft_consensus.cc:2258] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:04:14.789258 12891 tablet_replica.cc:330] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: stopping tablet replica
I20260810 18:04:14.789562 12891 raft_consensus.cc:2229] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:04:14.789961 12891 raft_consensus.cc:2258] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:04:14.788422 12892 raft_consensus.cc:2229] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:04:14.791554 12892 raft_consensus.cc:2258] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:04:14.791921 12891 ts_tablet_manager.cc:1905] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:04:14.793195 12892 ts_tablet_manager.cc:1905] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:04:14.794832 12891 ts_tablet_manager.cc:1918] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:04:14.794929 12891 log.cc:1196] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/b820ff37949e4886be1119a1ae1a7d9d
I20260810 18:04:14.795271 12891 ts_tablet_manager.cc:1939] T b820ff37949e4886be1119a1ae1a7d9d P c2d696c7201e4d84b764659b93456c2e: Deleting consensus metadata
I20260810 18:04:14.796052 12892 ts_tablet_manager.cc:1918] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:04:14.796152 12892 log.cc:1196] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/b820ff37949e4886be1119a1ae1a7d9d
I20260810 18:04:14.796514 12892 ts_tablet_manager.cc:1939] T b820ff37949e4886be1119a1ae1a7d9d P a58e77dd57524053b593d3db189d76a1: Deleting consensus metadata
I20260810 18:04:14.796672 12356 catalog_manager.cc:4841] TS c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429): tablet b820ff37949e4886be1119a1ae1a7d9d (table test-table [id=79e6d91faca84299a5a409c4b94cb9c4]) successfully deleted
I20260810 18:04:14.797669 12356 catalog_manager.cc:4841] TS a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221): tablet b820ff37949e4886be1119a1ae1a7d9d (table test-table [id=79e6d91faca84299a5a409c4b94cb9c4]) successfully deleted
W20260810 18:04:14.802744 12358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:43303 (error 108)
I20260810 18:04:14.803709 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 12576
I20260810 18:04:14.803889 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:14.803566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:43303 (local address 127.0.0.1:48756)
0810 18:04:14.803727 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:14.803801 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 18:04:14.805644 12358 catalog_manager.cc:4584] TS c68dfee61eea40509b6e6c38912b4837 (127.28.83.65:43303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b820ff37949e4886be1119a1ae1a7d9d: Network error: Client connection negotiation failed: client connection to 127.28.83.65:43303: connect: Connection refused (error 111)
I20260810 18:04:14.811568 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 12715
I20260810 18:04:14.818521 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 12338
I20260810 18:04:34.826241 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37985
--webserver_interface=127.28.83.126
--webserver_port=39579
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:04:34.941567 12894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:34.941928 12894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:34.942010 12894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:34.946370 12894 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 18:04:34.946460 12894 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 18:04:34.946489 12894 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 18:04:34.946521 12894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:04:34.946565 12894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:34.946612 12894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:34.946650 12894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:04:34.946686 12894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:04:34.952056 12894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:37985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=39579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:34.953243 12894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:34.954344 12894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:34.968357 12901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:34.968358 12900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:34.968358 12903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:34.968439 12894 server_base.cc:1034] running on GCE node
I20260810 18:04:34.969282 12894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:34.970479 12894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:34.971643 12894 hybrid_clock.cc:648] HybridClock initialized: now 1786385074971632 us; error 26 us; skew 500 ppm
Aug 10 18:04:34 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:34.978981 12894 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:04:34.979905 12894 webserver.cc:460] Webserver started at http://127.28.83.126:39579/ using document root <none> and password file <none>
I20260810 18:04:34.980465 12894 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:34.980540 12894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:34.984573 12894 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:04:34.986876 12910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:34.988168 12894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:04:34.988306 12894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "2611d4f76bf44d3c91d0173723804cf3"
format_stamp: "Formatted at 2026-08-10 18:04:13 on dist-test-slave-srgv"
server_key: "0f7f435462a1fa52fff193455598aac5"
server_key_iv: "5eb25f5a22dd0f86cd41fdcb1b822e58"
server_key_version: "encryptionkey@0"
I20260810 18:04:34.988920 12894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:35.013805 12894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:35.018128 12894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:35.018312 12894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:35.025599 12894 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:37985
I20260810 18:04:35.025899 12964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:37985 every 8 connection(s)
I20260810 18:04:35.026734 12894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:04:35.031164 12965 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:04:35.032832 12965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Bootstrap starting.
I20260810 18:04:35.034952 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 12894
I20260810 18:04:35.035477 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:43303
--local_ip_for_outbound_sockets=127.28.83.65
--tserver_master_addrs=127.28.83.126:37985
--webserver_port=34585
--webserver_interface=127.28.83.65
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:04:35.052428 12965 log.cc:824] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Log is configured to *not* fsync() on all Append() calls
I20260810 18:04:35.061705 12965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:04:35.062150 12965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Bootstrap complete.
I20260810 18:04:35.066481 12965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:35.066931 12965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2611d4f76bf44d3c91d0173723804cf3, State: Initialized, Role: FOLLOWER
I20260810 18:04:35.067445 12965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:35.067570 12965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:04:35.067678 12965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:04:35.067798 12965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:04:35.070012 12965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:35.070374 12965 leader_election.cc:302] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2611d4f76bf44d3c91d0173723804cf3; no voters:
I20260810 18:04:35.070745 12965 leader_election.cc:288] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:04:35.070834 12969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:04:35.071033 12969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 2 LEADER]: Becoming Leader. State: Replica: 2611d4f76bf44d3c91d0173723804cf3, State: Running, Role: LEADER
I20260810 18:04:35.071346 12969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:35.072024 12965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:04:35.072726 12970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2611d4f76bf44d3c91d0173723804cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } } }
I20260810 18:04:35.072870 12970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:04:35.073541 12971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2611d4f76bf44d3c91d0173723804cf3. Latest consensus state: current_term: 2 leader_uuid: "2611d4f76bf44d3c91d0173723804cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } } }
I20260810 18:04:35.073706 12971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:04:35.073933 12976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:04:35.079413 12976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:04:35.080317 12976 catalog_manager.cc:1259] Loaded cluster ID: 6e25583b5fe74c598fe461c4646150fa
I20260810 18:04:35.080420 12976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:04:35.083088 12976 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:04:35.083925 12976 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Loaded TSK: 0
I20260810 18:04:35.099151 12976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Generated new TSK 1
I20260810 18:04:35.099457 12976 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:04:35.170528 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:35.170897 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:35.170980 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:35.178458 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:35.178575 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:35.178682 12967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:04:35.185361 12967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:43303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=34585
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:35.186604 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:35.187633 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:35.197392 12993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:35.197778 12967 server_base.cc:1034] running on GCE node
W20260810 18:04:35.197386 12992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:35.197532 12996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:35.198572 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:35.199355 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:35.200559 12967 hybrid_clock.cc:648] HybridClock initialized: now 1786385075200524 us; error 45 us; skew 500 ppm
Aug 10 18:04:35 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:35.203974 12967 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:04:35.204967 12967 webserver.cc:460] Webserver started at http://127.28.83.65:34585/ using document root <none> and password file <none>
I20260810 18:04:35.205595 12967 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:35.205713 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:35.209743 12967 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 18:04:35.212018 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:35.213482 12967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:04:35.213627 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "c68dfee61eea40509b6e6c38912b4837"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "5c0488531102411f2bc45a9a0c857f9a"
server_key_iv: "e2bd9361c9b66eb8168d3fd45bd4981f"
server_key_version: "encryptionkey@0"
I20260810 18:04:35.214191 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:35.225926 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:35.229163 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:35.229369 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:35.230055 12967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:04:35.231204 13012 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:04:35.236752 12967 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:04:35.236860 12967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 18:04:35.236953 12967 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:04:35.238747 12967 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:04:35.238826 12967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 18:04:35.239110 13012 tablet_bootstrap.cc:492] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Bootstrap starting.
I20260810 18:04:35.250066 12967 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:43303
I20260810 18:04:35.251031 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:04:35.251814 13122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:43303 every 8 connection(s)
I20260810 18:04:35.253295 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 12967
I20260810 18:04:35.253834 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:40429
--local_ip_for_outbound_sockets=127.28.83.66
--tserver_master_addrs=127.28.83.126:37985
--webserver_port=46067
--webserver_interface=127.28.83.66
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:04:35.261574 13012 log.cc:824] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:04:35 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:04:35.270258 13012 tablet_bootstrap.cc:492] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:04:35.270776 13012 tablet_bootstrap.cc:492] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Bootstrap complete.
I20260810 18:04:35.272187 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.253179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:60341 (local address 127.28.83.126:37985)
0810 18:04:35.255471 (+ 2292us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.255511 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.255565 (+ 54us) server_negotiation.cc:407] Connection header received
0810 18:04:35.255855 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.255870 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.256283 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.256662 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:35.258310 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.260012 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.261767 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.262606 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.262813 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.266180 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:35.266187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:35.266202 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:35.266237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:35.268380 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.269031 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.269037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.269041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.269197 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.269634 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.269638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.269640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.270107 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:35.270275 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.271041 (+ 766us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.271342 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2173,"thread_start_us":111,"threads_started":1}
I20260810 18:04:35.272217 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.253191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:60341)
0810 18:04:35.254289 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:35.254330 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:35.255106 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:35.256773 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:35.256782 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:35.257277 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:04:35.258096 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.258118 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.260197 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.260201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.261541 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.261555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.262777 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.262781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.262978 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.263743 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:04:35.263763 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:04:35.265921 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:04:35.268518 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:35.268524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:35.268538 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:35.268889 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:35.269302 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:35.269305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:35.269307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:35.269512 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:35.270288 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:04:35.270293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:04:35.270651 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 18:04:35.270854 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 18:04:35.271204 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":177,"threads_started":1}
I20260810 18:04:35.272388 13012 ts_tablet_manager.cc:1397] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.009s
I20260810 18:04:35.275250 13123 heartbeater.cc:347] Connected to a master server at 127.28.83.126:37985
I20260810 18:04:35.275477 13123 heartbeater.cc:464] Registering TS with master...
I20260810 18:04:35.275367 13012 raft_consensus.cc:348] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.275877 13012 raft_consensus.cc:729] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68dfee61eea40509b6e6c38912b4837, State: Initialized, Role: FOLLOWER
I20260810 18:04:35.276023 13123 heartbeater.cc:511] Master 127.28.83.126:37985 requested a full tablet report, sending...
I20260810 18:04:35.276422 13012 consensus_queue.cc:260] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.277585 13012 ts_tablet_manager.cc:1428] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:04:35.278977 12927 ts_manager.cc:194] Registered new tserver with Master: c68dfee61eea40509b6e6c38912b4837 (127.28.83.65:43303)
I20260810 18:04:35.282907 12927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:60341
I20260810 18:04:35.285729 13123 heartbeater.cc:503] Master 127.28.83.126:37985 was elected leader, sending a full tablet report...
Aug 10 18:04:35 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.65@KRBTEST.COM
I20260810 18:04:35.293970 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.282224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:43303 (local address 127.0.0.1:60234)
0810 18:04:35.283043 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:35.283085 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:35.283212 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:35.284274 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:35.284280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:35.284657 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:04:35.284761 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.284768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.285786 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.285790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.286590 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.286610 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.287453 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.287458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.287563 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.288132 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:04:35.288148 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:04:35.289799 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:04:35.291921 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:35.291935 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:35.291945 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:35.292249 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:35.292577 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:35.292579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:35.292580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:35.292663 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:35.293272 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:04:35.293276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:04:35.293379 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 18:04:35.293574 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 18:04:35.293781 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":122,"threads_started":1}
I20260810 18:04:35.294006 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.282335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.28.83.65:43303)
0810 18:04:35.283818 (+ 1483us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.283830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.283851 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:04:35.283912 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.283917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.284118 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.284273 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:35.284895 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.285670 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.286728 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.287217 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.287326 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.289949 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:35.289959 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:35.289972 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:35.289999 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:35.291783 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.292372 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.292378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.292380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.292476 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.292758 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.292763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.292766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.293155 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:35.293305 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.293676 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.293839 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1386,"spinlock_wait_cycles":768,"thread_start_us":121,"threads_started":1}
I20260810 18:04:35.295084 13055 tablet_service.cc:1514] Processing DeleteTablet for tablet b820ff37949e4886be1119a1ae1a7d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:04:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:60234
I20260810 18:04:35.296128 13132 tablet_replica.cc:330] stopping tablet replica
I20260810 18:04:35.296470 13132 raft_consensus.cc:2229] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:04:35.296741 13132 raft_consensus.cc:2258] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:04:35.297787 13132 ts_tablet_manager.cc:1905] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:04:35.301259 13132 ts_tablet_manager.cc:1918] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:04:35.301379 13132 log.cc:1196] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal/wals/b820ff37949e4886be1119a1ae1a7d9d
I20260810 18:04:35.301695 13132 ts_tablet_manager.cc:1939] T b820ff37949e4886be1119a1ae1a7d9d P c68dfee61eea40509b6e6c38912b4837: Deleting consensus metadata
I20260810 18:04:35.303344 12914 catalog_manager.cc:4841] TS c68dfee61eea40509b6e6c38912b4837 (127.28.83.65:43303): tablet b820ff37949e4886be1119a1ae1a7d9d (table test-table [id=79e6d91faca84299a5a409c4b94cb9c4]) successfully deleted
W20260810 18:04:35.378641 13127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:35.378918 13127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:35.378993 13127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:35.383272 13127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:35.383342 13127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:35.383456 13127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:04:35.388844 13127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:40429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=46067
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:35.389950 13127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:35.390995 13127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:35.400950 13140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:35.401175 13127 server_base.cc:1034] running on GCE node
W20260810 18:04:35.400962 13137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:35.400950 13138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:35.401731 13127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:35.402402 13127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:35.403592 13127 hybrid_clock.cc:648] HybridClock initialized: now 1786385075403579 us; error 35 us; skew 500 ppm
Aug 10 18:04:35 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:35.406559 13127 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:04:35.407436 13127 webserver.cc:460] Webserver started at http://127.28.83.66:46067/ using document root <none> and password file <none>
I20260810 18:04:35.408005 13127 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:35.408133 13127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:35.412204 13127 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:04:35.414496 13147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:35.415863 13127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:04:35.416021 13127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "c2d696c7201e4d84b764659b93456c2e"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "6a60bee3e735320fe919413edb3df9fe"
server_key_iv: "15ea84207b6faee1f5a5e10accbe3b2d"
server_key_version: "encryptionkey@0"
I20260810 18:04:35.416800 13127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:35.434427 13127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:35.437458 13127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:35.437676 13127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:35.438313 13127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:04:35.439174 13127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:04:35.439250 13127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:35.439324 13127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:04:35.439368 13127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:35.448995 13127 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:40429
I20260810 18:04:35.449021 13266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:40429 every 8 connection(s)
I20260810 18:04:35.450068 13127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:04:35.452278 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 13127
I20260810 18:04:35.452715 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:40221
--local_ip_for_outbound_sockets=127.28.83.67
--tserver_master_addrs=127.28.83.126:37985
--webserver_port=37095
--webserver_interface=127.28.83.67
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:04:35 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:04:35.465400 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.452053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:33109 (local address 127.28.83.126:37985)
0810 18:04:35.452246 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.452251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.453123 (+ 872us) server_negotiation.cc:407] Connection header received
0810 18:04:35.454518 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.454521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.454579 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.454692 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:35.455865 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.456732 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.458089 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.458551 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.458697 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.461223 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:35.461229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:35.461236 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:35.461275 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:35.463097 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.463612 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.463618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.463629 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.463734 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.464102 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.464109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.464114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.464385 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:35.464508 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.465004 (+ 496us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.465168 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 18:04:35.466162 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.452472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:33109)
0810 18:04:35.452938 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:35.452991 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:35.454300 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:35.454871 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:35.454878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:35.455239 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:04:35.455705 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.455719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.456901 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.456904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.457948 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.457959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.458712 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.458715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.458860 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.459491 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:04:35.459508 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:04:35.461032 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:04:35.463220 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:35.463224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:35.463238 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:35.463506 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:35.463840 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:35.463842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:35.463843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:35.463985 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:35.464499 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:04:35.464504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:04:35.464749 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 18:04:35.464911 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 18:04:35.465180 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":137,"threads_started":1}
I20260810 18:04:35.467418 13267 heartbeater.cc:347] Connected to a master server at 127.28.83.126:37985
I20260810 18:04:35.467664 13267 heartbeater.cc:464] Registering TS with master...
I20260810 18:04:35.468179 13267 heartbeater.cc:511] Master 127.28.83.126:37985 requested a full tablet report, sending...
I20260810 18:04:35.469241 12927 ts_manager.cc:194] Registered new tserver with Master: c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429)
I20260810 18:04:35.469882 12927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:33109
W20260810 18:04:35.567145 13270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:35.567425 13270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:35.567468 13270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:35.571528 13270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:35.571583 13270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:35.571640 13270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:04:35.577243 13270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:40221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=37095
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:35.578462 13270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:35.579486 13270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:35.588884 13279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:35.588878 13277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:35.588878 13276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:35.589268 13270 server_base.cc:1034] running on GCE node
I20260810 18:04:35.589704 13270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:35.590380 13270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:35.591542 13270 hybrid_clock.cc:648] HybridClock initialized: now 1786385075591527 us; error 25 us; skew 500 ppm
Aug 10 18:04:35 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:35.594620 13270 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:04:35.595491 13270 webserver.cc:460] Webserver started at http://127.28.83.67:37095/ using document root <none> and password file <none>
I20260810 18:04:35.596057 13270 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:35.596182 13270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:35.600034 13270 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:04:35.602386 13286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:35.603596 13270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 18:04:35.603732 13270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "a58e77dd57524053b593d3db189d76a1"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "2fe63b8c9c0e6e57386cfce569b87ef0"
server_key_iv: "c67354a4aaccf29d2d52b3da5788e69a"
server_key_version: "encryptionkey@0"
I20260810 18:04:35.604323 13270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:35.624197 13270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:35.627331 13270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:35.627566 13270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:35.628185 13270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:04:35.629034 13270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:04:35.629106 13270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:35.629185 13270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:04:35.629230 13270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:35.638598 13270 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:40221
I20260810 18:04:35.638630 13404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:40221 every 8 connection(s)
I20260810 18:04:35.639590 13270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:04:35.641187 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 13270
Aug 10 18:04:35 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385075, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:04:35.651520 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.641960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.28.83.126:37985)
0810 18:04:35.642205 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.642211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.642229 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:04:35.642393 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.642399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.642458 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.642561 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:35.643626 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.644202 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.645454 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.645707 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.645788 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.647546 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:35.647549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:35.647551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:35.647573 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:35.649621 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.650128 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.650135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.650136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.650223 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.650484 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.650487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.650489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.650730 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:35.650900 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.651103 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.651203 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260810 18:04:35.655006 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.641670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:44717 (local address 127.28.83.126:37985)
0810 18:04:35.641821 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.641825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.642870 (+ 1045us) server_negotiation.cc:407] Connection header received
0810 18:04:35.644430 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.644432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.644470 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.644514 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:35.646062 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.646515 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.648016 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.648280 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.648359 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.651377 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:35.651380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:35.651383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:35.651407 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:35.653093 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.653756 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.653758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.653759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.653831 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.654147 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.654149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.654150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.654294 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:35.654365 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.654813 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.654902 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:04:35.655893 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.642111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:44717)
0810 18:04:35.642697 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:35.642758 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:35.644202 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:35.644724 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:35.644733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:35.645202 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:04:35.645910 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.645932 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.646647 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.646651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.647898 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.647909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.648412 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.648415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.648553 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.649245 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:04:35.649260 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:04:35.651201 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:04:35.653209 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:35.653213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:35.653224 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:35.653638 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:35.653911 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:35.653913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:35.653914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:35.654055 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:35.654379 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:04:35.654385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:04:35.654609 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 18:04:35.654756 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 18:04:35.655002 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":187,"threads_started":1}
I20260810 18:04:35.656994 13405 heartbeater.cc:347] Connected to a master server at 127.28.83.126:37985
I20260810 18:04:35.657250 13405 heartbeater.cc:464] Registering TS with master...
I20260810 18:04:35.657799 13405 heartbeater.cc:511] Master 127.28.83.126:37985 requested a full tablet report, sending...
I20260810 18:04:35.658767 12927 ts_manager.cc:194] Registered new tserver with Master: a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221)
I20260810 18:04:35.659313 12927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:44717
I20260810 18:04:35.665387 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:04:35.669811 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.666113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.28.83.126:37985)
0810 18:04:35.666277 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.666282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.666452 (+ 170us) server_negotiation.cc:407] Connection header received
0810 18:04:35.666647 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.666650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.666763 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.666878 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:04:35.667136 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.667672 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.668553 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.668786 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.668892 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.669211 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:04:35.669343 (+ 132us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 18:04:35.669674 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51936: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 18:04:35.669883 13125 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51936: token expired
W20260810 18:04:35.670243 12849 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.83.126:37985)
I20260810 18:04:35.677655 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.670620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.28.83.126:37985)
0810 18:04:35.670728 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.670732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.670750 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:04:35.670838 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.670840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.670873 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.670935 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:35.671572 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.672040 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.672917 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.673114 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.673195 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.674686 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:35.674689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:35.674691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:35.674712 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:35.676189 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.676647 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.676650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.676651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.676721 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:35.676953 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:35.676955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:35.676956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:35.677109 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:35.677217 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.677382 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.677532 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:04:35.680253 12927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:04:35.683560 12927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:04:35.691063 13055 tablet_service.cc:1467] Processing CreateTablet for tablet 65cb0af1ea424655af3632dadb471119 (DEFAULT_TABLE table=test-table [id=4866a7b3620347fdb37e88862ae866ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:04:35.691757 13055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cb0af1ea424655af3632dadb471119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:04:35.694728 13012 tablet_bootstrap.cc:492] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Bootstrap starting.
I20260810 18:04:35.696072 13012 tablet_bootstrap.cc:654] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Neither blocks nor log segments found. Creating new log.
I20260810 18:04:35.696219 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.690285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:40221 (local address 127.0.0.1:47250)
0810 18:04:35.690758 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:35.690772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:35.690868 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:35.691599 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:35.691601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:35.691628 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:04:35.691846 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.691853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.693187 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.693193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.694559 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.694572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.695733 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.695736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.695855 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.695900 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 18:04:35.695976 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:04:35.696048 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":181,"thread_start_us":99,"threads_started":1}
I20260810 18:04:35.696319 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.690041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:40429 (local address 127.0.0.1:43698)
0810 18:04:35.690682 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:35.690697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:35.690789 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:35.691547 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:35.691550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:35.691569 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:04:35.691711 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.691721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.693287 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.693290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.694317 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.694327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.695808 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.695812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.696026 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.696045 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:04:35.696114 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:04:35.696179 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":145}
I20260810 18:04:35.696712 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.690702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.28.83.67:40221)
0810 18:04:35.691134 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.691147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.691169 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:04:35.691244 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.691250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.691438 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.691708 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:04:35.691973 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.693068 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.694724 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.695592 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.695734 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.695886 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.696451 (+ 565us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.696553 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":112,"threads_started":1}
I20260810 18:04:35.697250 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.690249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.28.83.66:40429)
0810 18:04:35.691075 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.691085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.691105 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:04:35.691170 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.691175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.691385 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.691563 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:04:35.691919 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.693121 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.694448 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.695666 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.695861 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.696023 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.696241 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.697099 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":120,"threads_started":1}
I20260810 18:04:35.697378 13012 tablet_bootstrap.cc:492] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: No bootstrap required, opened a new log
I20260810 18:04:35.698102 13012 ts_tablet_manager.cc:1397] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260810 18:04:35.698554 13012 raft_consensus.cc:348] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.698673 13012 raft_consensus.cc:374] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:04:35.698710 13012 raft_consensus.cc:729] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68dfee61eea40509b6e6c38912b4837, State: Initialized, Role: FOLLOWER
I20260810 18:04:35.698863 13012 consensus_queue.cc:260] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.699141 13012 ts_tablet_manager.cc:1428] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:04:35.699771 13338 tablet_service.cc:1467] Processing CreateTablet for tablet 65cb0af1ea424655af3632dadb471119 (DEFAULT_TABLE table=test-table [id=4866a7b3620347fdb37e88862ae866ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:04:35.700021 13199 tablet_service.cc:1467] Processing CreateTablet for tablet 65cb0af1ea424655af3632dadb471119 (DEFAULT_TABLE table=test-table [id=4866a7b3620347fdb37e88862ae866ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:04:35.700968 13338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cb0af1ea424655af3632dadb471119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:04:35.701007 13199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cb0af1ea424655af3632dadb471119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:04:35.708029 13419 tablet_bootstrap.cc:492] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: Bootstrap starting.
I20260810 18:04:35.709074 13420 tablet_bootstrap.cc:492] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: Bootstrap starting.
I20260810 18:04:35.710920 13419 tablet_bootstrap.cc:654] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: Neither blocks nor log segments found. Creating new log.
I20260810 18:04:35.711660 13420 tablet_bootstrap.cc:654] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: Neither blocks nor log segments found. Creating new log.
I20260810 18:04:35.711934 13419 log.cc:824] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: Log is configured to *not* fsync() on all Append() calls
I20260810 18:04:35.712534 13420 log.cc:824] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:04:35.714046 13419 tablet_bootstrap.cc:492] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: No bootstrap required, opened a new log
I20260810 18:04:35.714262 13419 ts_tablet_manager.cc:1397] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:04:35.714707 13420 tablet_bootstrap.cc:492] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: No bootstrap required, opened a new log
I20260810 18:04:35.714895 13420 ts_tablet_manager.cc:1397] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:04:35.717314 13419 raft_consensus.cc:348] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.717664 13419 raft_consensus.cc:374] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:04:35.717744 13419 raft_consensus.cc:729] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58e77dd57524053b593d3db189d76a1, State: Initialized, Role: FOLLOWER
I20260810 18:04:35.717767 13420 raft_consensus.cc:348] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.718058 13420 raft_consensus.cc:374] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:04:35.718115 13420 raft_consensus.cc:729] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d696c7201e4d84b764659b93456c2e, State: Initialized, Role: FOLLOWER
I20260810 18:04:35.718235 13419 consensus_queue.cc:260] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.718518 13420 consensus_queue.cc:260] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.719029 13405 heartbeater.cc:503] Master 127.28.83.126:37985 was elected leader, sending a full tablet report...
I20260810 18:04:35.719290 13419 ts_tablet_manager.cc:1428] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:04:35.719503 13420 ts_tablet_manager.cc:1428] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:04:35.719841 13267 heartbeater.cc:503] Master 127.28.83.126:37985 was elected leader, sending a full tablet report...
W20260810 18:04:35.753307 13124 tablet.cc:2369] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:04:35.817667 13423 raft_consensus.cc:482] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:04:35.817961 13423 raft_consensus.cc:504] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.819288 13423 leader_election.cc:288] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429), c68dfee61eea40509b6e6c38912b4837 (127.28.83.65:43303)
I20260810 18:04:35.824134 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.819750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:58865 (local address 127.28.83.66:40429)
0810 18:04:35.819888 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.819892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.819910 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:04:35.819970 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.819973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.820038 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.820150 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:04:35.820441 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.821262 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.822900 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.823489 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.823630 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.823701 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.823911 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.823983 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:04:35.824137 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.819745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:45945 (local address 127.28.83.65:43303)
0810 18:04:35.820111 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.820117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.820134 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:04:35.820257 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.820259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.820319 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.820440 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:04:35.820735 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.821461 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.822900 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.823609 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.823718 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.823869 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.823908 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.823983 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260810 18:04:35.824184 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.819512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:40429 (local address 127.28.83.67:58865)
0810 18:04:35.819646 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:35.819664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:35.819761 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:35.820169 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:35.820172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:35.820187 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:04:35.820328 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.820334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.821366 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.821368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.822755 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.822767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.823625 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.823627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.823762 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.823792 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:04:35.823878 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 18:04:35.823994 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":50}
I20260810 18:04:35.824193 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.819626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:43303 (local address 127.28.83.67:45945)
0810 18:04:35.820034 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:35.820045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:35.820163 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:35.820470 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:35.820472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:35.820487 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:04:35.820607 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.820615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.821547 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.821550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.822755 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:35.822767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.823711 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.823716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:35.823803 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.823827 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:04:35.823886 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 18:04:35.823993 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":55,"thread_start_us":136,"threads_started":1}
I20260810 18:04:35.825062 13220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cb0af1ea424655af3632dadb471119" candidate_uuid: "a58e77dd57524053b593d3db189d76a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d696c7201e4d84b764659b93456c2e" is_pre_election: true
I20260810 18:04:35.825098 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cb0af1ea424655af3632dadb471119" candidate_uuid: "a58e77dd57524053b593d3db189d76a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68dfee61eea40509b6e6c38912b4837" is_pre_election: true
I20260810 18:04:35.825387 13220 raft_consensus.cc:2454] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a58e77dd57524053b593d3db189d76a1 in term 0.
I20260810 18:04:35.825387 13075 raft_consensus.cc:2454] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a58e77dd57524053b593d3db189d76a1 in term 0.
I20260810 18:04:35.825932 13290 leader_election.cc:302] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a58e77dd57524053b593d3db189d76a1, c68dfee61eea40509b6e6c38912b4837; no voters:
I20260810 18:04:35.826215 13423 raft_consensus.cc:2781] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:04:35.826320 13423 raft_consensus.cc:482] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:04:35.826380 13423 raft_consensus.cc:3037] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:04:35.827490 13423 raft_consensus.cc:504] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.827874 13423 leader_election.cc:288] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [CANDIDATE]: Term 1 election: Requested vote from peers c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429), c68dfee61eea40509b6e6c38912b4837 (127.28.83.65:43303)
I20260810 18:04:35.828248 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cb0af1ea424655af3632dadb471119" candidate_uuid: "a58e77dd57524053b593d3db189d76a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68dfee61eea40509b6e6c38912b4837"
I20260810 18:04:35.828270 13220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cb0af1ea424655af3632dadb471119" candidate_uuid: "a58e77dd57524053b593d3db189d76a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d696c7201e4d84b764659b93456c2e"
I20260810 18:04:35.828404 13220 raft_consensus.cc:3037] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:04:35.828413 13075 raft_consensus.cc:3037] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:04:35.829571 13220 raft_consensus.cc:2454] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a58e77dd57524053b593d3db189d76a1 in term 1.
I20260810 18:04:35.829586 13075 raft_consensus.cc:2454] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a58e77dd57524053b593d3db189d76a1 in term 1.
I20260810 18:04:35.829895 13290 leader_election.cc:302] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a58e77dd57524053b593d3db189d76a1, c68dfee61eea40509b6e6c38912b4837; no voters:
I20260810 18:04:35.830090 13423 raft_consensus.cc:2781] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:04:35.830319 13423 raft_consensus.cc:686] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 1 LEADER]: Becoming Leader. State: Replica: a58e77dd57524053b593d3db189d76a1, State: Running, Role: LEADER
I20260810 18:04:35.830642 13423 consensus_queue.cc:237] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:35.833734 12926 catalog_manager.cc:5495] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a58e77dd57524053b593d3db189d76a1 (127.28.83.67). New cstate: current_term: 1 leader_uuid: "a58e77dd57524053b593d3db189d76a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } health_report { overall_health: UNKNOWN } } }
I20260810 18:04:35.851249 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.847472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.28.83.67:40221)
0810 18:04:35.847635 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:35.847640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:35.847656 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:04:35.847766 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:35.847768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:35.847823 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:35.847938 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:04:35.848208 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.848736 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.849694 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:35.849916 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:35.850032 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:35.850376 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:04:35.850802 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:04:35.850944 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:35.851012 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 18:04:35.851072 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:04:35.857345 13075 raft_consensus.cc:1261] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Refusing update from remote peer a58e77dd57524053b593d3db189d76a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:04:35.857348 13220 raft_consensus.cc:1261] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [term 1 FOLLOWER]: Refusing update from remote peer a58e77dd57524053b593d3db189d76a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:04:35.858150 13427 consensus_queue.cc:1035] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:04:35.858351 13423 consensus_queue.cc:1035] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:04:35.864259 13431 mvcc.cc:205] Tried to move back new op lower bound from 7317033270703661056 to 7317033270604877824. Current Snapshot: MvccSnapshot[applied={T|T < 7317033270703661056}]
I20260810 18:04:35.873917 12927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:04:35.874110 12927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:04:35.876660 12927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Sending DeleteTablet for 3 replicas of tablet 65cb0af1ea424655af3632dadb471119
I20260810 18:04:35.877540 13055 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cb0af1ea424655af3632dadb471119 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:04:35 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:60234
I20260810 18:04:35.877575 13338 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cb0af1ea424655af3632dadb471119 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:04:35 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:47250
I20260810 18:04:35.877579 13199 tablet_service.cc:1514] Processing DeleteTablet for tablet 65cb0af1ea424655af3632dadb471119 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:04:35 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:43698
I20260810 18:04:35.878165 13442 tablet_replica.cc:330] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: stopping tablet replica
I20260810 18:04:35.878451 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 12967
I20260810 18:04:35.878448 13441 tablet_replica.cc:330] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: stopping tablet replica
I20260810 18:04:35.878537 13442 raft_consensus.cc:2229] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:04:35.878726 13441 raft_consensus.cc:2229] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:04:35.878895 13442 raft_consensus.cc:2258] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:04:35.879436 13443 tablet_replica.cc:330] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: stopping tablet replica
I20260810 18:04:35.879820 13443 raft_consensus.cc:2229] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:04:35.880366 13443 raft_consensus.cc:2258] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:04:35.882411 13442 ts_tablet_manager.cc:1905] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:04:35.882902 13443 ts_tablet_manager.cc:1905] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:04:35.885386 13442 ts_tablet_manager.cc:1918] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:04:35.885596 13442 log.cc:1196] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/65cb0af1ea424655af3632dadb471119
I20260810 18:04:35.886063 13442 ts_tablet_manager.cc:1939] T 65cb0af1ea424655af3632dadb471119 P c2d696c7201e4d84b764659b93456c2e: Deleting consensus metadata
I20260810 18:04:35.886734 13443 ts_tablet_manager.cc:1918] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:04:35.886839 13443 log.cc:1196] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/65cb0af1ea424655af3632dadb471119
I20260810 18:04:35.887207 13443 ts_tablet_manager.cc:1939] T 65cb0af1ea424655af3632dadb471119 P a58e77dd57524053b593d3db189d76a1: Deleting consensus metadata
I20260810 18:04:35.887665 12912 catalog_manager.cc:4841] TS c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429): tablet 65cb0af1ea424655af3632dadb471119 (table test-table [id=4866a7b3620347fdb37e88862ae866ec]) successfully deleted
I20260810 18:04:35.888537 12912 catalog_manager.cc:4841] TS a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221): tablet 65cb0af1ea424655af3632dadb471119 (table test-table [id=4866a7b3620347fdb37e88862ae866ec]) successfully deleted
W20260810 18:04:35.892140 12914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:43303 (error 108)
I20260810 18:04:35.892764 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 13127
I20260810 18:04:35.895087 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:35.892946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:43303 (local address 127.0.0.1:60240)
0810 18:04:35.894883 (+ 1937us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:35.894972 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:43303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1821}
W20260810 18:04:35.897634 12914 catalog_manager.cc:4584] TS c68dfee61eea40509b6e6c38912b4837 (127.28.83.65:43303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65cb0af1ea424655af3632dadb471119: Network error: Client connection negotiation failed: client connection to 127.28.83.65:43303: connect: Connection refused (error 111)
I20260810 18:04:35.900758 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 13270
I20260810 18:04:35.907838 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 12894
I20260810 18:04:55.915553 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37985
--webserver_interface=127.28.83.126
--webserver_port=39579
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:04:56.030925 13445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:56.031188 13445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:56.031272 13445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:56.035995 13445 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 18:04:56.036057 13445 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 18:04:56.036073 13445 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 18:04:56.036087 13445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:04:56.036106 13445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:56.036161 13445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:56.036190 13445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:04:56.036219 13445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:04:56.041823 13445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:37985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=39579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:56.042945 13445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:56.044005 13445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:04:56.053725 13445 server_base.cc:1034] running on GCE node
W20260810 18:04:56.053747 13450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:56.053747 13451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:56.053927 13453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:56.054616 13445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:56.055845 13445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:56.057049 13445 hybrid_clock.cc:648] HybridClock initialized: now 1786385096057044 us; error 36 us; skew 500 ppm
Aug 10 18:04:56 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385096, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:56.060155 13445 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:04:56.061192 13445 webserver.cc:460] Webserver started at http://127.28.83.126:39579/ using document root <none> and password file <none>
I20260810 18:04:56.061858 13445 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:56.061959 13445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:56.066031 13445 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:04:56.068434 13460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:56.069782 13445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:04:56.069936 13445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "2611d4f76bf44d3c91d0173723804cf3"
format_stamp: "Formatted at 2026-08-10 18:04:13 on dist-test-slave-srgv"
server_key: "0f7f435462a1fa52fff193455598aac5"
server_key_iv: "5eb25f5a22dd0f86cd41fdcb1b822e58"
server_key_version: "encryptionkey@0"
I20260810 18:04:56.070533 13445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:56.087484 13445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:56.091203 13445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:56.091426 13445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:56.098917 13445 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:37985
I20260810 18:04:56.099215 13515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:37985 every 8 connection(s)
I20260810 18:04:56.100052 13445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:04:56.103111 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 13445
I20260810 18:04:56.103542 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:43303
--local_ip_for_outbound_sockets=127.28.83.65
--tserver_master_addrs=127.28.83.126:37985
--webserver_port=34585
--webserver_interface=127.28.83.65
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:04:56.104425 13516 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:04:56.106105 13516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Bootstrap starting.
I20260810 18:04:56.124919 13516 log.cc:824] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Log is configured to *not* fsync() on all Append() calls
I20260810 18:04:56.136924 13516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:04:56.137385 13516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Bootstrap complete.
I20260810 18:04:56.142045 13516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:56.142495 13516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2611d4f76bf44d3c91d0173723804cf3, State: Initialized, Role: FOLLOWER
I20260810 18:04:56.143009 13516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:56.143153 13516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:04:56.143247 13516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:04:56.143373 13516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:04:56.145510 13516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:56.145850 13516 leader_election.cc:302] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2611d4f76bf44d3c91d0173723804cf3; no voters:
I20260810 18:04:56.146211 13516 leader_election.cc:288] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 18:04:56.146317 13521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:04:56.146510 13521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [term 3 LEADER]: Becoming Leader. State: Replica: 2611d4f76bf44d3c91d0173723804cf3, State: Running, Role: LEADER
I20260810 18:04:56.146806 13521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } }
I20260810 18:04:56.147483 13516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:04:56.148166 13522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2611d4f76bf44d3c91d0173723804cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } } }
I20260810 18:04:56.148303 13522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:04:56.148857 13523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2611d4f76bf44d3c91d0173723804cf3. Latest consensus state: current_term: 3 leader_uuid: "2611d4f76bf44d3c91d0173723804cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2611d4f76bf44d3c91d0173723804cf3" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37985 } } }
I20260810 18:04:56.148947 13523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:04:56.149210 13528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:04:56.154312 13528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:04:56.155035 13528 catalog_manager.cc:1259] Loaded cluster ID: 6e25583b5fe74c598fe461c4646150fa
I20260810 18:04:56.155120 13528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:04:56.156867 13528 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:04:56.157568 13528 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Loaded TSK: 0
I20260810 18:04:56.157655 13528 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2611d4f76bf44d3c91d0173723804cf3: Loaded TSK: 1
F20260810 18:04:56.158272 13528 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f59e2df19ad google::LogMessage::Fail() at ??:0
@ 0x7f59e2df488a google::LogMessage::SendToLog() at ??:0
@ 0x7f59e2df14de google::LogMessage::Flush() at ??:0
@ 0x7f59e2df2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f59e5ab518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f59e5aadf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f59ed75a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f59ed73499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f59ed735178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f59ed776df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f59ed79582b std::function<>::operator()() at ??:0
@ 0x7f59ed734b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f59ed735f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f59ed732579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f59ed776640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f59e50433d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f59e5043c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f59e50453f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f59e5034512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f59e66786db start_thread at ??:0
@ 0x7f59e18ef61f clone at ??:0
W20260810 18:04:56.245137 13518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:56.245388 13518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:56.245501 13518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:56.249948 13518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:56.249998 13518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:56.250061 13518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:04:56.255923 13518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:43303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=34585
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:56.256979 13518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:56.258229 13518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:56.268384 13565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:56.268474 13567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:56.268553 13563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:56.269075 13518 server_base.cc:1034] running on GCE node
I20260810 18:04:56.269511 13518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:56.270123 13518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:56.271272 13518 hybrid_clock.cc:648] HybridClock initialized: now 1786385096271240 us; error 43 us; skew 500 ppm
Aug 10 18:04:56 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385096, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:56.274096 13518 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:04:56.275004 13518 webserver.cc:460] Webserver started at http://127.28.83.65:34585/ using document root <none> and password file <none>
I20260810 18:04:56.275574 13518 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:56.275668 13518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:56.279812 13518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 18:04:56.282229 13575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:56.283594 13518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:04:56.283733 13518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "c68dfee61eea40509b6e6c38912b4837"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "5c0488531102411f2bc45a9a0c857f9a"
server_key_iv: "e2bd9361c9b66eb8168d3fd45bd4981f"
server_key_version: "encryptionkey@0"
I20260810 18:04:56.284315 13518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:56.309700 13518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:56.313025 13518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:56.313241 13518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:56.313928 13518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:04:56.315062 13582 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:04:56.321019 13518 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:04:56.321161 13518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 18:04:56.321259 13518 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:04:56.323132 13518 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:04:56.323221 13518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 18:04:56.323536 13582 tablet_bootstrap.cc:492] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Bootstrap starting.
I20260810 18:04:56.334784 13518 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:43303
I20260810 18:04:56.334813 13694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:43303 every 8 connection(s)
I20260810 18:04:56.335882 13518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:04:56.342119 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 13518
I20260810 18:04:56.342574 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:40429
--local_ip_for_outbound_sockets=127.28.83.66
--tserver_master_addrs=127.28.83.126:37985
--webserver_port=46067
--webserver_interface=127.28.83.66
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:04:56.347607 13582 log.cc:824] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Log is configured to *not* fsync() on all Append() calls
I20260810 18:04:56.356909 13582 tablet_bootstrap.cc:492] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:04:56.357366 13582 tablet_bootstrap.cc:492] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Bootstrap complete.
I20260810 18:04:56.359020 13582 ts_tablet_manager.cc:1397] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Time spent bootstrapping tablet: real 0.036s user 0.021s sys 0.012s
I20260810 18:04:56.362649 13582 raft_consensus.cc:348] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:56.363222 13582 raft_consensus.cc:729] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68dfee61eea40509b6e6c38912b4837, State: Initialized, Role: FOLLOWER
I20260810 18:04:56.363749 13582 consensus_queue.cc:260] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:56.364822 13582 ts_tablet_manager.cc:1428] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260810 18:04:56.416216 13697 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:04:56.338005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:33909)
0810 18:04:56.338526 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:56.338589 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:56.339432 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:56.415139 (+ 75707us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":160,"threads_started":1}
W20260810 18:04:56.416968 13695 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:37985 (0 consecutive failures): Network error: Failed to ping master at 127.28.83.126:37985: Client connection negotiation failed: client connection to 127.28.83.126:37985: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 18:04:56.417833 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:56.417529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:44069)
0810 18:04:56.417673 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:56.417764 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 18:04:56.418540 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:56.418304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:46733)
0810 18:04:56.418423 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:56.418474 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 18:04:56.418744 13695 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:04:56.461059 13698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:56.461331 13698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:56.461400 13698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:56.465766 13698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:56.465826 13698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:56.465883 13698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:04:56.471134 13698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:40429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=46067
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:56.472196 13698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:56.473238 13698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:56.483220 13707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:56.483207 13708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:56.483260 13698 server_base.cc:1034] running on GCE node
W20260810 18:04:56.483207 13710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:56.483934 13698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:56.484604 13698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:56.485793 13698 hybrid_clock.cc:648] HybridClock initialized: now 1786385096485781 us; error 27 us; skew 500 ppm
Aug 10 18:04:56 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385096, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:56.488636 13698 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:04:56.489634 13698 webserver.cc:460] Webserver started at http://127.28.83.66:46067/ using document root <none> and password file <none>
I20260810 18:04:56.490223 13698 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:56.490324 13698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:56.494536 13698 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:04:56.496697 13717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:56.498026 13698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:04:56.498185 13698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "c2d696c7201e4d84b764659b93456c2e"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "6a60bee3e735320fe919413edb3df9fe"
server_key_iv: "15ea84207b6faee1f5a5e10accbe3b2d"
server_key_version: "encryptionkey@0"
I20260810 18:04:56.498775 13698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:56.531352 13698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:56.534749 13698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:56.534992 13698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:56.535647 13698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:04:56.536518 13698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:04:56.536592 13698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:56.536666 13698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:04:56.536702 13698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:56.545938 13698 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:40429
I20260810 18:04:56.545965 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:40429 every 8 connection(s)
I20260810 18:04:56.546972 13698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:04:56.550169 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 13698
I20260810 18:04:56.550616 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:40221
--local_ip_for_outbound_sockets=127.28.83.67
--tserver_master_addrs=127.28.83.126:37985
--webserver_port=37095
--webserver_interface=127.28.83.67
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:04:56.551095 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:56.549151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:34273)
0810 18:04:56.549665 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:56.549775 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":124,"threads_started":1}
W20260810 18:04:56.551558 13837 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:37985 (0 consecutive failures): Network error: Failed to ping master at 127.28.83.126:37985: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
I20260810 18:04:56.552192 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:56.551983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:59753)
0810 18:04:56.552094 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:56.552140 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 18:04:56.552834 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:56.552625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:47795)
0810 18:04:56.552734 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:56.552787 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:04:56.553021 13837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:04:56.664415 13840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:04:56.664701 13840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:04:56.664747 13840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:04:56.668870 13840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:04:56.668934 13840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:04:56.669056 13840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:04:56.674489 13840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:40221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=37095
--tserver_master_addrs=127.28.83.126:37985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:04:56.675643 13840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:04:56.676688 13840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:04:56.686582 13848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:56.686582 13846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:04:56.686646 13845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:04:56.686780 13840 server_base.cc:1034] running on GCE node
I20260810 18:04:56.687415 13840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:04:56.688110 13840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:04:56.689302 13840 hybrid_clock.cc:648] HybridClock initialized: now 1786385096689276 us; error 50 us; skew 500 ppm
Aug 10 18:04:56 dist-test-slave-srgv krb5kdc[12325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385096, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:04:56.692137 13840 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:04:56.693025 13840 webserver.cc:460] Webserver started at http://127.28.83.67:37095/ using document root <none> and password file <none>
I20260810 18:04:56.693665 13840 fs_manager.cc:359] Metadata directory not provided
I20260810 18:04:56.693761 13840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:04:56.697655 13840 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:04:56.699790 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:56.701077 13840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:04:56.701217 13840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "a58e77dd57524053b593d3db189d76a1"
format_stamp: "Formatted at 2026-08-10 18:04:14 on dist-test-slave-srgv"
server_key: "2fe63b8c9c0e6e57386cfce569b87ef0"
server_key_iv: "c67354a4aaccf29d2d52b3da5788e69a"
server_key_version: "encryptionkey@0"
I20260810 18:04:56.701910 13840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:04:56.732923 13840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:04:56.736508 13840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:04:56.736737 13840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:04:56.737383 13840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:04:56.738296 13840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:04:56.738365 13840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:56.738441 13840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:04:56.738495 13840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:04:56.747968 13840 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:40221
I20260810 18:04:56.747997 13973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:40221 every 8 connection(s)
I20260810 18:04:56.748974 13840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:04:56.752849 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:56.751108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:52791)
0810 18:04:56.751566 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:56.751676 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
W20260810 18:04:56.753295 13974 heartbeater.cc:644] Failed to heartbeat to 127.28.83.126:37985 (0 consecutive failures): Network error: Failed to ping master at 127.28.83.126:37985: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
I20260810 18:04:56.753933 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:56.753704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:42867)
0810 18:04:56.753825 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:56.753875 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 18:04:56.754500 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:56.754297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:41795)
0810 18:04:56.754402 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:56.754446 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 18:04:56.754673 13974 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:04:56.758638 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 13840
I20260810 18:04:57.420010 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:57.419504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:45427)
0810 18:04:57.419809 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:57.419878 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
I20260810 18:04:57.506568 13980 raft_consensus.cc:482] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:04:57.506798 13980 raft_consensus.cc:504] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:57.507959 13980 leader_election.cc:288] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429), a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221)
Aug 10 18:04:57 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385096, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.67@KRBTEST.COM
Aug 10 18:04:57 dist-test-slave-srgv krb5kdc[12325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385096, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.66@KRBTEST.COM
I20260810 18:04:57.523823 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:57.508185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:40429 (local address 127.28.83.65:55719)
0810 18:04:57.508328 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:57.508346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:57.508457 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:57.510292 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:57.510298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:57.510863 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:04:57.511365 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:57.511389 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:57.512692 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:57.512698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:57.514256 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:57.514269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:57.515259 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:57.515263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:57.515553 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:57.516593 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:04:57.516611 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:04:57.519037 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:04:57.521194 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:57.521197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:57.521199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:57.521554 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:57.521938 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:57.521942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:57.521944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:57.522111 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:57.522785 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:04:57.522792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:04:57.523066 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 18:04:57.523257 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 18:04:57.523598 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":414,"spinlock_wait_cycles":10752}
I20260810 18:04:57.523833 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:57.508362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:55719 (local address 127.28.83.66:40429)
0810 18:04:57.508759 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:57.508772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:57.508810 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:04:57.509565 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:57.509572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:57.509936 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:57.510257 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:57.511556 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:57.512541 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:57.514392 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:57.515106 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:57.515282 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:57.519257 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:57.519262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:57.519277 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:57.519307 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:57.521056 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:57.521680 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:57.521693 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:57.521697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:57.521830 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:57.522223 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:57.522231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:57.522232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:57.522659 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:57.522816 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:57.523393 (+ 577us) server_negotiation.cc:299] Negotiation successful
0810 18:04:57.523649 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":82,"threads_started":1}
I20260810 18:04:57.523874 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:57.508666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:36337 (local address 127.28.83.67:40221)
0810 18:04:57.509082 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 18:04:57.509092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:04:57.509125 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:04:57.509807 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:04:57.509813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:04:57.510110 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:04:57.510384 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:04:57.511544 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:57.512587 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:57.514392 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:04:57.515106 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:04:57.515282 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:57.518935 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:04:57.518940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:04:57.518953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:04:57.518982 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:04:57.520988 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:57.521690 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:57.521703 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:57.521706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:57.521830 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:04:57.522228 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:04:57.522231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:04:57.522233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:04:57.522660 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:04:57.522809 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:04:57.523388 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 18:04:57.523669 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":137,"threads_started":1}
I20260810 18:04:57.524374 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:57.508482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:40221 (local address 127.28.83.65:36337)
0810 18:04:57.508752 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:57.508767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:04:57.508876 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:04:57.510341 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:04:57.510344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:04:57.510868 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:04:57.511364 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:57.511380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:57.512749 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:57.512752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:57.514264 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:04:57.514269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:04:57.515271 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:04:57.515274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:04:57.515551 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:04:57.516568 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:04:57.516592 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:04:57.518684 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:04:57.521130 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:57.521136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:57.521150 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:57.521568 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:57.521942 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:04:57.521945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:04:57.521947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:04:57.522119 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:04:57.522789 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:04:57.522792 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:04:57.523065 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 18:04:57.523259 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 18:04:57.524129 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":277,"spinlock_wait_cycles":31232,"thread_start_us":82,"threads_started":1}
W20260810 18:04:57.527588 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
W20260810 18:04:57.527773 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
I20260810 18:04:57.527819 13577 leader_election.cc:302] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c68dfee61eea40509b6e6c38912b4837; no voters: a58e77dd57524053b593d3db189d76a1, c2d696c7201e4d84b764659b93456c2e
I20260810 18:04:57.528049 13980 raft_consensus.cc:2726] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:04:57.554122 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:57.553661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:47423)
0810 18:04:57.553947 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:57.554007 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260810 18:04:57.755755 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:57.755316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:42283)
0810 18:04:57.755612 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:57.755662 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
I20260810 18:04:58.421296 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:58.420868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:56299)
0810 18:04:58.421124 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:58.421186 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260810 18:04:58.555395 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:58.554953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:53519)
0810 18:04:58.555215 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:58.555289 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260810 18:04:58.756866 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:58.756448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:42705)
0810 18:04:58.756704 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:58.756757 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
I20260810 18:04:59.215269 13989 raft_consensus.cc:482] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:04:59.215472 13989 raft_consensus.cc:504] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:04:59.215968 13989 leader_election.cc:288] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429), a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221)
W20260810 18:04:59.216995 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
W20260810 18:04:59.217108 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
I20260810 18:04:59.217160 13577 leader_election.cc:302] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c68dfee61eea40509b6e6c38912b4837; no voters: a58e77dd57524053b593d3db189d76a1, c2d696c7201e4d84b764659b93456c2e
I20260810 18:04:59.217306 13989 raft_consensus.cc:2726] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:04:59.422546 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:59.422088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:57083)
0810 18:04:59.422370 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:59.422442 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
I20260810 18:04:59.556663 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:59.556213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:34757)
0810 18:04:59.556503 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:59.556568 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260810 18:04:59.758461 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:04:59.757807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:48787)
0810 18:04:59.758253 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 18:04:59.758327 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":96,"threads_started":1}
I20260810 18:05:00.423847 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:00.423404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:43875)
0810 18:05:00.423678 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:00.423749 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":114,"threads_started":1}
I20260810 18:05:00.558010 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:00.557582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:56113)
0810 18:05:00.557847 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:00.557910 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260810 18:05:00.759953 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:00.759491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:52631)
0810 18:05:00.759793 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:00.759856 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":125,"threads_started":1}
I20260810 18:05:01.425529 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:01.424945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:52361)
0810 18:05:01.425280 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:01.425345 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":97,"threads_started":1}
I20260810 18:05:01.559321 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:01.558880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:56363)
0810 18:05:01.559173 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:01.559223 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":130,"threads_started":1}
I20260810 18:05:01.761291 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:01.760863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:56029)
0810 18:05:01.761128 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:01.761191 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
I20260810 18:05:02.323278 14000 raft_consensus.cc:482] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:02.323443 14000 raft_consensus.cc:504] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:05:02.323817 14000 leader_election.cc:288] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429), a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221)
W20260810 18:05:02.324694 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
W20260810 18:05:02.324864 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
I20260810 18:05:02.324931 13577 leader_election.cc:302] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c68dfee61eea40509b6e6c38912b4837; no voters: a58e77dd57524053b593d3db189d76a1, c2d696c7201e4d84b764659b93456c2e
I20260810 18:05:02.325152 14000 raft_consensus.cc:2726] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:05:02.426803 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:02.426317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:39375)
0810 18:05:02.426630 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:02.426700 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260810 18:05:02.560765 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:02.560282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:41619)
0810 18:05:02.560578 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:02.560646 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260810 18:05:02.762558 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:02.762122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:60181)
0810 18:05:02.762399 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:02.762455 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260810 18:05:03.428218 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:03.427752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:46217)
0810 18:05:03.428048 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:03.428115 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260810 18:05:03.562299 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:03.561740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:52189)
0810 18:05:03.562131 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:03.562197 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":102,"threads_started":1}
I20260810 18:05:03.763913 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:03.763415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:49069)
0810 18:05:03.763733 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:03.763813 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":78,"threads_started":1}
I20260810 18:05:04.429656 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:04.429142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:43743)
0810 18:05:04.429478 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:04.429543 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260810 18:05:04.563593 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:04.563149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:52539)
0810 18:05:04.563427 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:04.563492 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260810 18:05:04.629164 14009 raft_consensus.cc:482] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:04.629319 14009 raft_consensus.cc:504] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:05:04.629871 14009 leader_election.cc:288] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429), a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221)
W20260810 18:05:04.630751 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
W20260810 18:05:04.630910 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
I20260810 18:05:04.630985 13577 leader_election.cc:302] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c68dfee61eea40509b6e6c38912b4837; no voters: a58e77dd57524053b593d3db189d76a1, c2d696c7201e4d84b764659b93456c2e
I20260810 18:05:04.631165 14009 raft_consensus.cc:2726] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:05:04.765177 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:04.764742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:34825)
0810 18:05:04.765020 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:04.765080 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260810 18:05:05.430934 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:05.430496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:43191)
0810 18:05:05.430767 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:05.430833 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260810 18:05:05.564930 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:05.564456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:40783)
0810 18:05:05.564748 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:05.564814 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260810 18:05:05.766543 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:05.766071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:36069)
0810 18:05:05.766351 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:05.766441 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260810 18:05:06.432305 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:06.431879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:44973)
0810 18:05:06.432148 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:06.432208 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260810 18:05:06.566193 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:06.565760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:60489)
0810 18:05:06.566034 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:06.566092 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260810 18:05:06.767884 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:06.767414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:55957)
0810 18:05:06.767717 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:06.767785 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260810 18:05:07.433768 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:07.433261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:37745)
0810 18:05:07.433601 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:07.433663 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":98,"threads_started":1}
I20260810 18:05:07.567452 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:07.566982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:41355)
0810 18:05:07.567300 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:07.567354 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":128,"threads_started":1}
I20260810 18:05:07.769276 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:07.768815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:33245)
0810 18:05:07.769115 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:07.769177 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260810 18:05:08.435079 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:08.434619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:37731)
0810 18:05:08.434909 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:08.434975 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260810 18:05:08.568848 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:08.568343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:60103)
0810 18:05:08.568681 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:08.568749 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":97,"threads_started":1}
I20260810 18:05:08.770431 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:08.770004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:34587)
0810 18:05:08.770279 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:08.770336 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260810 18:05:08.779443 14024 raft_consensus.cc:482] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:08.779701 14024 raft_consensus.cc:504] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:05:08.780115 14024 leader_election.cc:288] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429), a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221)
W20260810 18:05:08.780998 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
W20260810 18:05:08.781204 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
I20260810 18:05:08.781261 13577 leader_election.cc:302] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c68dfee61eea40509b6e6c38912b4837; no voters: a58e77dd57524053b593d3db189d76a1, c2d696c7201e4d84b764659b93456c2e
I20260810 18:05:08.781498 14024 raft_consensus.cc:2726] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:05:09.436462 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:09.435968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:49399)
0810 18:05:09.436288 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:09.436358 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
I20260810 18:05:09.570364 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:09.569889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:50221)
0810 18:05:09.570196 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:09.570262 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
I20260810 18:05:09.771797 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:09.771309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:53985)
0810 18:05:09.771624 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:09.771698 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
I20260810 18:05:10.330463 14028 raft_consensus.cc:482] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:10.330638 14028 raft_consensus.cc:504] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d696c7201e4d84b764659b93456c2e" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 40429 } } peers { permanent_uuid: "a58e77dd57524053b593d3db189d76a1" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 40221 } } peers { permanent_uuid: "c68dfee61eea40509b6e6c38912b4837" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 43303 } }
I20260810 18:05:10.331092 14028 leader_election.cc:288] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429), a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221)
W20260810 18:05:10.331949 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2d696c7201e4d84b764659b93456c2e (127.28.83.66:40429): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
W20260810 18:05:10.332142 13577 leader_election.cc:341] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a58e77dd57524053b593d3db189d76a1 (127.28.83.67:40221): Not found: Tablet not found: 65cb0af1ea424655af3632dadb471119
I20260810 18:05:10.332252 13577 leader_election.cc:302] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c68dfee61eea40509b6e6c38912b4837; no voters: a58e77dd57524053b593d3db189d76a1, c2d696c7201e4d84b764659b93456c2e
I20260810 18:05:10.332442 14028 raft_consensus.cc:2726] T 65cb0af1ea424655af3632dadb471119 P c68dfee61eea40509b6e6c38912b4837 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:05:10.437834 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:10.437332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:34573)
0810 18:05:10.437665 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:10.437729 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":149,"threads_started":1}
I20260810 18:05:10.571693 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:10.571236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:47997)
0810 18:05:10.571526 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:10.571591 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260810 18:05:10.773303 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:10.772720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:44977)
0810 18:05:10.773097 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:10.773179 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":88,"threads_started":1}
I20260810 18:05:11.439293 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:11.438812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.65:37217)
0810 18:05:11.439123 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:11.439192 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260810 18:05:11.573364 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:11.572872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.66:51747)
0810 18:05:11.573198 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:11.573264 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":87,"threads_started":1}
I20260810 18:05:11.769160 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 13518
I20260810 18:05:11.774716 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:11.774286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:37985 (local address 127.28.83.67:60747)
0810 18:05:11.774555 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:11.774618 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.126:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
I20260810 18:05:11.776474 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 13698
I20260810 18:05:11.782882 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 13840
2026-08-10T18:05:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59389 ms)
[----------] 30 tests from SecurityITest (247934 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:11 dist-test-slave-srgv krb5kdc[14038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:11 dist-test-slave-srgv krb5kdc[14038](info): set up 2 sockets
Aug 10 18:05:11 dist-test-slave-srgv krb5kdc[14038](info): commencing operation
krb5kdc: starting...
W20260810 18:05:12.845383 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:12 dist-test-slave-srgv krb5kdc[14038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:12.974831 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44789
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:44789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:13.087118 14054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:13.087443 14054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:13.087522 14054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:13.091703 14054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:13.091811 14054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:13.091847 14054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:13.091890 14054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:13.091923 14054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:13.097579 14054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:44789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:13.098707 14054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:13.099764 14054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:13.108757 14059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:13.108860 14063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:13.108837 14054 server_base.cc:1034] running on GCE node
W20260810 18:05:13.108841 14060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:13.109687 14054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:13.110755 14054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:13.111923 14054 hybrid_clock.cc:648] HybridClock initialized: now 1786385113111898 us; error 39 us; skew 500 ppm
Aug 10 18:05:13 dist-test-slave-srgv krb5kdc[14038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:13.114778 14054 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:13.115686 14054 webserver.cc:460] Webserver started at http://127.28.83.126:39725/ using document root <none> and password file <none>
I20260810 18:05:13.116210 14054 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:13.116269 14054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:13.116437 14054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:13.118552 14054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "a6eb0076f4724674880e2457befeaae0"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "633e1f783d30237122b20d165b8f767e"
server_key_iv: "ca5834194f942eccbf2c44283d339a05"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.119081 14054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "a6eb0076f4724674880e2457befeaae0"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "633e1f783d30237122b20d165b8f767e"
server_key_iv: "ca5834194f942eccbf2c44283d339a05"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.122753 14054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:05:13.125079 14070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.126428 14054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:05:13.126530 14054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "a6eb0076f4724674880e2457befeaae0"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "633e1f783d30237122b20d165b8f767e"
server_key_iv: "ca5834194f942eccbf2c44283d339a05"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.126619 14054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:13.143318 14054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:13.146415 14054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:13.146720 14054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:13.154779 14054 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:44789
I20260810 18:05:13.154793 14129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:44789 every 8 connection(s)
I20260810 18:05:13.155740 14054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:13.158567 14130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:13.161986 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 14054
I20260810 18:05:13.162153 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:13.162446 29005 external_mini_cluster.cc:1428] Setting key 49143552171a095b0898273c71a55c54
I20260810 18:05:13.164788 14130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0: Bootstrap starting.
I20260810 18:05:13.167207 14130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:13.167996 14130 log.cc:824] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:13.170264 14130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0: No bootstrap required, opened a new log
Aug 10 18:05:13 dist-test-slave-srgv krb5kdc[14038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:13.173692 14130 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6eb0076f4724674880e2457befeaae0" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44789 } }
I20260810 18:05:13.173938 14130 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:13.173998 14130 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6eb0076f4724674880e2457befeaae0, State: Initialized, Role: FOLLOWER
I20260810 18:05:13.174458 14130 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6eb0076f4724674880e2457befeaae0" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44789 } }
I20260810 18:05:13.174588 14130 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:13.174675 14130 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:13.174777 14130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:13.175671 14130 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6eb0076f4724674880e2457befeaae0" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44789 } }
I20260810 18:05:13.176028 14130 leader_election.cc:302] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6eb0076f4724674880e2457befeaae0; no voters:
I20260810 18:05:13.176335 14130 leader_election.cc:288] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:13.176450 14136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:13.176679 14136 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [term 1 LEADER]: Becoming Leader. State: Replica: a6eb0076f4724674880e2457befeaae0, State: Running, Role: LEADER
I20260810 18:05:13.177054 14136 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6eb0076f4724674880e2457befeaae0" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44789 } }
I20260810 18:05:13.177644 14130 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:13.178670 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.163937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.28.83.126:44789)
0810 18:05:13.164412 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:13.164427 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:13.164461 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:05:13.165153 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:13.165171 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:13.165563 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:13.165887 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:13.166749 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.167835 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.168925 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.169385 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.169568 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.172386 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:13.172391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:13.172403 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:13.172430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:13.174949 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.175400 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.175406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.175411 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.175532 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.175836 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.175840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.175841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.176302 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:13.176503 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:13.177348 (+ 845us) server_negotiation.cc:299] Negotiation successful
0810 18:05:13.177667 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":143,"threads_started":1}
I20260810 18:05:13.178804 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6eb0076f4724674880e2457befeaae0. Latest consensus state: current_term: 1 leader_uuid: "a6eb0076f4724674880e2457befeaae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6eb0076f4724674880e2457befeaae0" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44789 } } }
I20260810 18:05:13.179167 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:13.179025 14138 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6eb0076f4724674880e2457befeaae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6eb0076f4724674880e2457befeaae0" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44789 } } }
I20260810 18:05:13.179477 14138 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0 [sys.catalog]: This master's current role is: LEADER
W20260810 18:05:13.182801 14153 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:13.182902 14153 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:05:13.183024 14147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:13.183719 14147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:13.189497 14147 catalog_manager.cc:1347] Generated new cluster ID: fbdadbcf628e491e8677de89ac1c5227
I20260810 18:05:13.189574 14147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:13.211807 14147 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:13.213085 14147 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:13.225323 14147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0: Generated new TSK 0
I20260810 18:05:13.226200 14147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:05:13.265298 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44789
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:13.381484 14159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:13.381760 14159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:13.381806 14159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:13.385957 14159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:13.386029 14159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:13.386088 14159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:05:13.391621 14159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:13.392726 14159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:13.393810 14159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:13.403801 14169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:13.403847 14166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:13.403847 14165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:13.404093 14159 server_base.cc:1034] running on GCE node
I20260810 18:05:13.404597 14159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:13.405241 14159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:13.406411 14159 hybrid_clock.cc:648] HybridClock initialized: now 1786385113406392 us; error 40 us; skew 500 ppm
Aug 10 18:05:13 dist-test-slave-srgv krb5kdc[14038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:13.409340 14159 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:05:13.410323 14159 webserver.cc:460] Webserver started at http://127.28.83.65:41197/ using document root <none> and password file <none>
I20260810 18:05:13.410912 14159 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:13.410976 14159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:13.411199 14159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:13.413322 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "0fb01d517d104e4f904c8154d58919c5"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "087c4e4d563f84cb482ec661b5855690"
server_key_iv: "002181eb09c50ae41a83a5d77f902217"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.413987 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "0fb01d517d104e4f904c8154d58919c5"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "087c4e4d563f84cb482ec661b5855690"
server_key_iv: "002181eb09c50ae41a83a5d77f902217"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.417955 14159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:05:13.420528 14176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.421936 14159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:13.422091 14159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "0fb01d517d104e4f904c8154d58919c5"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "087c4e4d563f84cb482ec661b5855690"
server_key_iv: "002181eb09c50ae41a83a5d77f902217"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.422221 14159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:13.433933 14159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:13.437371 14159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:13.437650 14159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:13.438279 14159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:13.439199 14159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:13.439273 14159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.439347 14159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:13.439405 14159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.448866 14159 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:35429
I20260810 18:05:13.448894 14296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:35429 every 8 connection(s)
I20260810 18:05:13.449867 14159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:05:13.452688 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 14159
I20260810 18:05:13.452843 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:05:13.453202 29005 external_mini_cluster.cc:1428] Setting key 225664677c15aee16204ec4b9faf7cba
Aug 10 18:05:13 dist-test-slave-srgv krb5kdc[14038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:05:13.465209 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.451745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:34475 (local address 127.28.83.126:44789)
0810 18:05:13.451896 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:13.451899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:13.453132 (+ 1233us) server_negotiation.cc:407] Connection header received
0810 18:05:13.454193 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:13.454196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:13.454252 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:13.454356 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:13.455944 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.456526 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.457783 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.458032 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.458118 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.461187 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:13.461193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:13.461198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:13.461236 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:13.463023 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.463538 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.463542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.463543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.463657 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.463997 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.464000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.464002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.464265 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:13.464387 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:13.464960 (+ 573us) server_negotiation.cc:299] Negotiation successful
0810 18:05:13.465072 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:05:13.466149 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.452127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44789 (local address 127.28.83.65:34475)
0810 18:05:13.452934 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:13.452997 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:13.453970 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:13.454527 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:13.454534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:13.455060 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:13.455775 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.455798 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.456695 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.456698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.457642 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.457652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.458206 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.458213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.458448 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.459122 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:13.459148 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:13.461000 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:13.463148 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:13.463153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:13.463166 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:13.463430 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:13.463759 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:13.463761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:13.463762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:13.463897 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:13.464404 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:13.464469 (+ 65us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:13.464726 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 18:05:13.464893 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 18:05:13.465185 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
I20260810 18:05:13.467638 14297 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44789
I20260810 18:05:13.467898 14297 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:13.468412 14297 heartbeater.cc:511] Master 127.28.83.126:44789 requested a full tablet report, sending...
I20260810 18:05:13.470175 14089 ts_manager.cc:194] Registered new tserver with Master: 0fb01d517d104e4f904c8154d58919c5 (127.28.83.65:35429)
I20260810 18:05:13.471846 14089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:34475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:05:13.484396 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44789
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:13.599722 14302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:13.600020 14302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:13.600067 14302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:13.604254 14302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:13.604332 14302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:13.604393 14302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:05:13.610016 14302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:13.611266 14302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:13.612350 14302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:13.622252 14309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:13.622243 14308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:13.622243 14311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:13.622464 14302 server_base.cc:1034] running on GCE node
I20260810 18:05:13.623050 14302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:13.623689 14302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:13.624886 14302 hybrid_clock.cc:648] HybridClock initialized: now 1786385113624865 us; error 32 us; skew 500 ppm
Aug 10 18:05:13 dist-test-slave-srgv krb5kdc[14038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:13.628253 14302 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:05:13.629257 14302 webserver.cc:460] Webserver started at http://127.28.83.66:45333/ using document root <none> and password file <none>
I20260810 18:05:13.629894 14302 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:13.629962 14302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:13.630195 14302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:13.632462 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "3745b46879ba47e38ec99bc1f4a15b70"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "e52bbfb110cb5379aa2a202b8ac4b587"
server_key_iv: "123bc52ec3e31c0ece085fa6aac2efcc"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.633072 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "3745b46879ba47e38ec99bc1f4a15b70"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "e52bbfb110cb5379aa2a202b8ac4b587"
server_key_iv: "123bc52ec3e31c0ece085fa6aac2efcc"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.637099 14302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 18:05:13.639808 14318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.641124 14302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 18:05:13.641271 14302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "3745b46879ba47e38ec99bc1f4a15b70"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "e52bbfb110cb5379aa2a202b8ac4b587"
server_key_iv: "123bc52ec3e31c0ece085fa6aac2efcc"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.641399 14302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:13.671799 14302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:13.675364 14302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:13.675585 14302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:13.676267 14302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:13.677130 14302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:13.677179 14302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.677220 14302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:13.677278 14302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.686823 14302 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:34927
I20260810 18:05:13.686843 14436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:34927 every 8 connection(s)
I20260810 18:05:13.687769 14302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:05:13.692265 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 14302
I20260810 18:05:13.692425 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:05:13.692703 29005 external_mini_cluster.cc:1428] Setting key cf01959b3ae1795380000a01a0ee9fad
Aug 10 18:05:13 dist-test-slave-srgv krb5kdc[14038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:13.703287 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.689604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:46059 (local address 127.28.83.126:44789)
0810 18:05:13.689754 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:13.689759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:13.690860 (+ 1101us) server_negotiation.cc:407] Connection header received
0810 18:05:13.691778 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:13.691781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:13.691841 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:13.691948 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:13.693372 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.694161 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.695922 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.696139 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.696291 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.699066 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:13.699072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:13.699076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:13.699104 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:13.700896 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.701451 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.701454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.701456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.701556 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.701923 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.701928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.701932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.702201 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:13.702321 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:13.702878 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 18:05:13.703036 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
I20260810 18:05:13.704327 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.690007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44789 (local address 127.28.83.66:46059)
0810 18:05:13.690691 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:13.690734 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:13.691569 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:13.692122 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:13.692129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:13.692622 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:13.693176 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.693200 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.694311 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.694314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.695787 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.695803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.696362 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.696370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.696577 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.697237 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:13.697255 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:13.698873 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:13.701021 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:13.701026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:13.701040 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:13.701303 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:13.701658 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:13.701660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:13.701662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:13.701817 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:13.702330 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:13.702335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:13.702592 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 18:05:13.702778 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 18:05:13.703174 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":110,"threads_started":1}
I20260810 18:05:13.705747 14438 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44789
I20260810 18:05:13.706055 14438 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:05:13.706665 14438 heartbeater.cc:511] Master 127.28.83.126:44789 requested a full tablet report, sending...
I20260810 18:05:13.707850 14089 ts_manager.cc:194] Registered new tserver with Master: 3745b46879ba47e38ec99bc1f4a15b70 (127.28.83.66:34927)
I20260810 18:05:13.708540 14089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:46059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:05:13.724730 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44789
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:13.840746 14443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:13.841040 14443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:13.841095 14443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:13.845463 14443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:13.845551 14443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:13.845654 14443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:05:13.851454 14443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:13.852553 14443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:13.853609 14443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:13.863572 14448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:13.863582 14452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:13.863582 14449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:13.863785 14443 server_base.cc:1034] running on GCE node
I20260810 18:05:13.864344 14443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:13.865031 14443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:13.866232 14443 hybrid_clock.cc:648] HybridClock initialized: now 1786385113866171 us; error 75 us; skew 500 ppm
Aug 10 18:05:13 dist-test-slave-srgv krb5kdc[14038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:13.869307 14443 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:05:13.870355 14443 webserver.cc:460] Webserver started at http://127.28.83.67:41005/ using document root <none> and password file <none>
I20260810 18:05:13.870956 14443 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:13.871055 14443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:13.871331 14443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:13.873656 14443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "61ac6be7389342209b4911cd00464671"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "3dd996632c2af616bdd2916bf71848b8"
server_key_iv: "0dca1ee79664484488fc3f4adfed9fee"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.874280 14443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "61ac6be7389342209b4911cd00464671"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "3dd996632c2af616bdd2916bf71848b8"
server_key_iv: "0dca1ee79664484488fc3f4adfed9fee"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.878219 14443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:05:13.880774 14459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.882113 14443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:05:13.882277 14443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "61ac6be7389342209b4911cd00464671"
format_stamp: "Formatted at 2026-08-10 18:05:13 on dist-test-slave-srgv"
server_key: "3dd996632c2af616bdd2916bf71848b8"
server_key_iv: "0dca1ee79664484488fc3f4adfed9fee"
server_key_version: "encryptionkey@0"
I20260810 18:05:13.882393 14443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:13.904999 14443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:13.908324 14443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:13.908545 14443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:13.909180 14443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:13.910051 14443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:13.910133 14443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.910236 14443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:13.910288 14443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:13.920048 14443 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:38605
I20260810 18:05:13.920123 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:38605 every 8 connection(s)
I20260810 18:05:13.920994 14443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:05:13.922744 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 14443
I20260810 18:05:13.922894 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:05:13.923226 29005 external_mini_cluster.cc:1428] Setting key 17f3bc490600dc3c97f8bb41dd326292
Aug 10 18:05:13 dist-test-slave-srgv krb5kdc[14038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385113, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:13.935534 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.923194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:47725 (local address 127.28.83.126:44789)
0810 18:05:13.923353 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:13.923357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:13.924539 (+ 1182us) server_negotiation.cc:407] Connection header received
0810 18:05:13.925689 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:13.925691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:13.925740 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:13.925813 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:13.927045 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.927627 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.928740 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.928975 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.929066 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.931794 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:13.931797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:13.931799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:13.931827 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:13.933548 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.934072 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.934075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.934076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.934156 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.934475 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.934477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.934478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.934679 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:13.934763 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:13.935283 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 18:05:13.935399 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:05:13.936658 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.923618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44789 (local address 127.28.83.67:47725)
0810 18:05:13.924365 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:13.924421 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:13.925467 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:13.926025 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:13.926035 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:13.926426 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:13.926919 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.926931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.927749 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.927751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.928614 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.928632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.929116 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.929119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.929276 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.929952 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:13.929968 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:13.931616 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:13.933674 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:13.933680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:13.933692 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:13.933976 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:13.934244 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:13.934246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:13.934247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:13.934389 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:13.934789 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:13.934794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:13.935039 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 18:05:13.935220 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 18:05:13.935490 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":139,"threads_started":1}
I20260810 18:05:13.937831 14582 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44789
I20260810 18:05:13.938102 14582 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:13.938633 14582 heartbeater.cc:511] Master 127.28.83.126:44789 requested a full tablet report, sending...
I20260810 18:05:13.939749 14088 ts_manager.cc:194] Registered new tserver with Master: 61ac6be7389342209b4911cd00464671 (127.28.83.67:38605)
I20260810 18:05:13.940392 14088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:47725
I20260810 18:05:13.950039 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:05:13.960108 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.952332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.28.83.126:44789)
0810 18:05:13.952572 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:13.952577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:13.952737 (+ 160us) server_negotiation.cc:407] Connection header received
0810 18:05:13.952911 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:13.952914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:13.952957 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:13.953043 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:13.953862 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.954389 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.955244 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.955460 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.955571 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.957027 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:13.957030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:13.957033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:13.957072 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:13.958438 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.958923 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.958926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.958927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.958999 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:13.959269 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:13.959271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:13.959272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:13.959488 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:13.959591 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:13.959812 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 18:05:13.959909 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260810 18:05:13.963107 14088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:05:13.966174 14088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:05:13.981068 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.975262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.28.83.66:34927)
0810 18:05:13.975695 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:13.975709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:13.975734 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:05:13.975815 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:13.975823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:13.976022 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:13.976480 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:13.976648 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.977707 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.979358 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.980316 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.980429 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.980580 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:13.980788 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 18:05:13.980899 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":108,"threads_started":1}
I20260810 18:05:13.981055 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.974530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:34927 (local address 127.0.0.1:40238)
0810 18:05:13.974949 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:13.975009 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:13.975169 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:13.976204 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:13.976213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:13.976240 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:13.976460 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.976469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.977833 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.977838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.979230 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.979240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.980454 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.980458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.980565 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.980617 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 18:05:13.980736 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 18:05:13.980871 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":54,"thread_start_us":116,"threads_started":1}
I20260810 18:05:13.981388 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.974877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.28.83.65:35429)
0810 18:05:13.975241 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:13.975251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:13.975626 (+ 375us) server_negotiation.cc:407] Connection header received
0810 18:05:13.975712 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:13.975717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:13.975921 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:13.976065 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:13.976620 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.977866 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.979536 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.980316 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.980429 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.980580 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:13.981127 (+ 547us) server_negotiation.cc:299] Negotiation successful
0810 18:05:13.981249 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":68,"threads_started":1}
I20260810 18:05:13.981755 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.975472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:38605 (local address 127.0.0.1:44706)
0810 18:05:13.976059 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:13.976075 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:13.976171 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:13.976872 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:13.976876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:13.976897 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:13.976991 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.976996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.978475 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.978478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.979617 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.979624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.980663 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.980666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.980845 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.980865 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:05:13.980929 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 18:05:13.981638 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":663,"thread_start_us":111,"threads_started":1}
I20260810 18:05:13.981942 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.975920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.28.83.67:38605)
0810 18:05:13.976408 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:13.976420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:13.976443 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:05:13.976525 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:13.976533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:13.976726 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:13.976873 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:13.977115 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.978331 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.979731 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.980336 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.980459 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.980600 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:13.981016 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 18:05:13.981663 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":171,"threads_started":1}
I20260810 18:05:13.982035 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:13.974548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:35429 (local address 127.0.0.1:59442)
0810 18:05:13.975420 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:13.975433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:13.975518 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:13.976247 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:13.976253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:13.976275 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:13.976478 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.976487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.978028 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.978031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.979414 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:13.979424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:13.980647 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:13.980650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:13.980824 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:13.980861 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 18:05:13.980931 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:05:13.981016 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":111,"thread_start_us":112,"threads_started":1}
I20260810 18:05:13.984392 14512 tablet_service.cc:1467] Processing CreateTablet for tablet 5422f27c777d4c0a8c51570bb66c7ed1 (DEFAULT_TABLE table=test-table [id=1c68840ccdbb468abc27fcbe27d4c44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:13.984582 14226 tablet_service.cc:1467] Processing CreateTablet for tablet 5422f27c777d4c0a8c51570bb66c7ed1 (DEFAULT_TABLE table=test-table [id=1c68840ccdbb468abc27fcbe27d4c44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:13.984436 14366 tablet_service.cc:1467] Processing CreateTablet for tablet 5422f27c777d4c0a8c51570bb66c7ed1 (DEFAULT_TABLE table=test-table [id=1c68840ccdbb468abc27fcbe27d4c44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:13.985564 14226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5422f27c777d4c0a8c51570bb66c7ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:13.985564 14366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5422f27c777d4c0a8c51570bb66c7ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:13.985579 14512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5422f27c777d4c0a8c51570bb66c7ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:13.992172 14600 tablet_bootstrap.cc:492] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: Bootstrap starting.
I20260810 18:05:13.993708 14601 tablet_bootstrap.cc:492] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5: Bootstrap starting.
I20260810 18:05:13.993708 14602 tablet_bootstrap.cc:492] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: Bootstrap starting.
I20260810 18:05:13.994508 14600 tablet_bootstrap.cc:654] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:13.995589 14600 log.cc:824] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:13.995894 14601 tablet_bootstrap.cc:654] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:13.996302 14602 tablet_bootstrap.cc:654] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:13.996807 14601 log.cc:824] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:13.997090 14602 log.cc:824] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:13.998570 14600 tablet_bootstrap.cc:492] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: No bootstrap required, opened a new log
I20260810 18:05:13.998803 14600 ts_tablet_manager.cc:1397] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 18:05:13.998870 14602 tablet_bootstrap.cc:492] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: No bootstrap required, opened a new log
I20260810 18:05:13.999060 14602 ts_tablet_manager.cc:1397] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 18:05:13.999882 14601 tablet_bootstrap.cc:492] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5: No bootstrap required, opened a new log
I20260810 18:05:14.000065 14601 ts_tablet_manager.cc:1397] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 18:05:14.002734 14602 raft_consensus.cc:348] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } }
I20260810 18:05:14.003046 14602 raft_consensus.cc:374] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:14.003098 14602 raft_consensus.cc:729] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3745b46879ba47e38ec99bc1f4a15b70, State: Initialized, Role: FOLLOWER
I20260810 18:05:14.003289 14600 raft_consensus.cc:348] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } }
I20260810 18:05:14.003545 14600 raft_consensus.cc:374] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:14.003626 14600 raft_consensus.cc:729] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ac6be7389342209b4911cd00464671, State: Initialized, Role: FOLLOWER
I20260810 18:05:14.003595 14602 consensus_queue.cc:260] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } }
I20260810 18:05:14.004037 14600 consensus_queue.cc:260] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } }
I20260810 18:05:14.004735 14602 ts_tablet_manager.cc:1428] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260810 18:05:14.005038 14600 ts_tablet_manager.cc:1428] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:05:14.004914 14601 raft_consensus.cc:348] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } }
I20260810 18:05:14.005103 14438 heartbeater.cc:503] Master 127.28.83.126:44789 was elected leader, sending a full tablet report...
I20260810 18:05:14.005162 14601 raft_consensus.cc:374] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:14.005218 14601 raft_consensus.cc:729] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb01d517d104e4f904c8154d58919c5, State: Initialized, Role: FOLLOWER
I20260810 18:05:14.005362 14582 heartbeater.cc:503] Master 127.28.83.126:44789 was elected leader, sending a full tablet report...
I20260810 18:05:14.005636 14601 consensus_queue.cc:260] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } }
I20260810 18:05:14.006520 14601 ts_tablet_manager.cc:1428] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 18:05:14.006850 14297 heartbeater.cc:503] Master 127.28.83.126:44789 was elected leader, sending a full tablet report...
I20260810 18:05:14.086903 14608 raft_consensus.cc:482] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:14.087234 14608 raft_consensus.cc:504] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } }
I20260810 18:05:14.088465 14608 leader_election.cc:288] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3745b46879ba47e38ec99bc1f4a15b70 (127.28.83.66:34927), 61ac6be7389342209b4911cd00464671 (127.28.83.67:38605)
I20260810 18:05:14.092669 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.088847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:42489 (local address 127.28.83.66:34927)
0810 18:05:14.088986 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:14.088991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:14.089452 (+ 461us) server_negotiation.cc:407] Connection header received
0810 18:05:14.089589 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:14.089592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:14.089646 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:14.089733 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:14.090022 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.090524 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.091652 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.092061 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.092189 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.092246 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:14.092449 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 18:05:14.092527 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:05:14.092692 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.088778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:38605 (local address 127.28.83.65:43807)
0810 18:05:14.089170 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:14.089183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:14.089302 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:14.089587 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:14.089590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:14.089608 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:14.089768 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:14.089775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.090611 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.090616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:14.091515 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:14.091525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.092233 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.092237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:14.092344 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.092362 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:05:14.092460 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 18:05:14.092527 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260810 18:05:14.092687 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.088881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:43807 (local address 127.28.83.67:38605)
0810 18:05:14.089032 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:14.089037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:14.089282 (+ 245us) server_negotiation.cc:407] Connection header received
0810 18:05:14.089421 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:14.089425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:14.089479 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:14.089559 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:14.089902 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.090484 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.091658 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.092083 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.092232 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.092288 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:14.092503 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 18:05:14.092583 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:05:14.092702 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.088777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:34927 (local address 127.28.83.65:42489)
0810 18:05:14.089295 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:14.089311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:14.089453 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:14.089753 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:14.089755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:14.089771 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:14.089902 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:14.089909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.090691 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.090693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:14.091519 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:14.091528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.092196 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.092200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:14.092296 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.092326 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:05:14.092400 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 18:05:14.092496 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":57,"thread_start_us":125,"threads_started":1}
I20260810 18:05:14.093510 14534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5422f27c777d4c0a8c51570bb66c7ed1" candidate_uuid: "0fb01d517d104e4f904c8154d58919c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ac6be7389342209b4911cd00464671" is_pre_election: true
I20260810 18:05:14.093701 14387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5422f27c777d4c0a8c51570bb66c7ed1" candidate_uuid: "0fb01d517d104e4f904c8154d58919c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3745b46879ba47e38ec99bc1f4a15b70" is_pre_election: true
I20260810 18:05:14.093852 14534 raft_consensus.cc:2454] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fb01d517d104e4f904c8154d58919c5 in term 0.
I20260810 18:05:14.094043 14387 raft_consensus.cc:2454] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fb01d517d104e4f904c8154d58919c5 in term 0.
I20260810 18:05:14.094468 14179 leader_election.cc:302] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb01d517d104e4f904c8154d58919c5, 61ac6be7389342209b4911cd00464671; no voters:
I20260810 18:05:14.094791 14608 raft_consensus.cc:2781] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:05:14.094882 14608 raft_consensus.cc:482] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:05:14.094925 14608 raft_consensus.cc:3037] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:14.095986 14608 raft_consensus.cc:504] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } }
I20260810 18:05:14.096336 14608 leader_election.cc:288] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [CANDIDATE]: Term 1 election: Requested vote from peers 3745b46879ba47e38ec99bc1f4a15b70 (127.28.83.66:34927), 61ac6be7389342209b4911cd00464671 (127.28.83.67:38605)
I20260810 18:05:14.096755 14534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5422f27c777d4c0a8c51570bb66c7ed1" candidate_uuid: "0fb01d517d104e4f904c8154d58919c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ac6be7389342209b4911cd00464671"
I20260810 18:05:14.096870 14534 raft_consensus.cc:3037] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:14.098254 14534 raft_consensus.cc:2454] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb01d517d104e4f904c8154d58919c5 in term 1.
I20260810 18:05:14.098555 14387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5422f27c777d4c0a8c51570bb66c7ed1" candidate_uuid: "0fb01d517d104e4f904c8154d58919c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3745b46879ba47e38ec99bc1f4a15b70"
I20260810 18:05:14.098601 14179 leader_election.cc:302] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb01d517d104e4f904c8154d58919c5, 61ac6be7389342209b4911cd00464671; no voters:
I20260810 18:05:14.098714 14387 raft_consensus.cc:3037] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:14.098829 14608 raft_consensus.cc:2781] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:14.099066 14608 raft_consensus.cc:686] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [term 1 LEADER]: Becoming Leader. State: Replica: 0fb01d517d104e4f904c8154d58919c5, State: Running, Role: LEADER
I20260810 18:05:14.099447 14608 consensus_queue.cc:237] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } }
I20260810 18:05:14.100137 14387 raft_consensus.cc:2454] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb01d517d104e4f904c8154d58919c5 in term 1.
I20260810 18:05:14.102913 14088 catalog_manager.cc:5495] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fb01d517d104e4f904c8154d58919c5 (127.28.83.65). New cstate: current_term: 1 leader_uuid: "0fb01d517d104e4f904c8154d58919c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fb01d517d104e4f904c8154d58919c5" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 35429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 } health_report { overall_health: UNKNOWN } } }
I20260810 18:05:14.136389 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.132859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.28.83.65:35429)
0810 18:05:14.133055 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:14.133059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:14.133080 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:05:14.133199 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:14.133202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:14.133296 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:14.133463 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:14.133717 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.134214 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.135030 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.135242 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.135315 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.135594 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:14.136007 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:14.136100 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:14.136153 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 18:05:14.136204 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 18:05:14.142556 14387 raft_consensus.cc:1261] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [term 1 FOLLOWER]: Refusing update from remote peer 0fb01d517d104e4f904c8154d58919c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:14.142556 14534 raft_consensus.cc:1261] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [term 1 FOLLOWER]: Refusing update from remote peer 0fb01d517d104e4f904c8154d58919c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:14.143285 14608 consensus_queue.cc:1035] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61ac6be7389342209b4911cd00464671" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:14.143474 14612 consensus_queue.cc:1035] T 5422f27c777d4c0a8c51570bb66c7ed1 P 0fb01d517d104e4f904c8154d58919c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3745b46879ba47e38ec99bc1f4a15b70" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 34927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:14.157351 14616 mvcc.cc:205] Tried to move back new op lower bound from 7317033427520479232 to 7317033427354927104. Current Snapshot: MvccSnapshot[applied={T|T < 7317033427520479232 or (T in {7317033427520479232})}]
I20260810 18:05:14.161576 14617 mvcc.cc:205] Tried to move back new op lower bound from 7317033427520479232 to 7317033427354927104. Current Snapshot: MvccSnapshot[applied={T|T < 7317033427520479232 or (T in {7317033427520479232})}]
I20260810 18:05:14.163075 14089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:14.163247 14089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:14.165866 14089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6eb0076f4724674880e2457befeaae0: Sending DeleteTablet for 3 replicas of tablet 5422f27c777d4c0a8c51570bb66c7ed1
I20260810 18:05:14.166682 14512 tablet_service.cc:1514] Processing DeleteTablet for tablet 5422f27c777d4c0a8c51570bb66c7ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:44706
I20260810 18:05:14.166749 14366 tablet_service.cc:1514] Processing DeleteTablet for tablet 5422f27c777d4c0a8c51570bb66c7ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:40238
I20260810 18:05:14.167232 14226 tablet_service.cc:1514] Processing DeleteTablet for tablet 5422f27c777d4c0a8c51570bb66c7ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:14 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:59442
I20260810 18:05:14.167407 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 14159
I20260810 18:05:14.169858 14624 tablet_replica.cc:330] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: stopping tablet replica
I20260810 18:05:14.170222 14624 raft_consensus.cc:2229] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:14.170571 14624 raft_consensus.cc:2258] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:14.171020 14625 tablet_replica.cc:330] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: stopping tablet replica
I20260810 18:05:14.171340 14625 raft_consensus.cc:2229] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:14.171645 14625 raft_consensus.cc:2258] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:14.172634 14624 ts_tablet_manager.cc:1905] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:14.173189 14625 ts_tablet_manager.cc:1905] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:14.175822 14624 ts_tablet_manager.cc:1918] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:14.175917 14625 ts_tablet_manager.cc:1918] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:14.176019 14625 log.cc:1196] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/5422f27c777d4c0a8c51570bb66c7ed1
I20260810 18:05:14.176055 14624 log.cc:1196] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/5422f27c777d4c0a8c51570bb66c7ed1
I20260810 18:05:14.176405 14625 ts_tablet_manager.cc:1939] T 5422f27c777d4c0a8c51570bb66c7ed1 P 61ac6be7389342209b4911cd00464671: Deleting consensus metadata
I20260810 18:05:14.176596 14624 ts_tablet_manager.cc:1939] T 5422f27c777d4c0a8c51570bb66c7ed1 P 3745b46879ba47e38ec99bc1f4a15b70: Deleting consensus metadata
W20260810 18:05:14.176841 14071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:35429 (error 108)
I20260810 18:05:14.177811 14075 catalog_manager.cc:4841] TS 3745b46879ba47e38ec99bc1f4a15b70 (127.28.83.66:34927): tablet 5422f27c777d4c0a8c51570bb66c7ed1 (table test-table [id=1c68840ccdbb468abc27fcbe27d4c44f]) successfully deleted
I20260810 18:05:14.177870 14073 catalog_manager.cc:4841] TS 61ac6be7389342209b4911cd00464671 (127.28.83.67:38605): tablet 5422f27c777d4c0a8c51570bb66c7ed1 (table test-table [id=1c68840ccdbb468abc27fcbe27d4c44f]) successfully deleted
I20260810 18:05:14.178092 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.177694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:35429 (local address 127.0.0.1:59454)
0810 18:05:14.177909 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:14.177989 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:35429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260810 18:05:14.178252 14071 catalog_manager.cc:4584] TS 0fb01d517d104e4f904c8154d58919c5 (127.28.83.65:35429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5422f27c777d4c0a8c51570bb66c7ed1: Network error: Client connection negotiation failed: client connection to 127.28.83.65:35429: connect: Connection refused (error 111)
I20260810 18:05:14.178581 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 14302
I20260810 18:05:14.187143 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 14443
I20260810 18:05:14.194233 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 14054
2026-08-10T18:05:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2411 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T18:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:14Z Disabled control of system clock
I20260810 18:05:14.227243 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43489
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:33191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:43489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:05:14.339530 14630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:14.339828 14630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:14.339864 14630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:14.343832 14630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:14.343888 14630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:14.343904 14630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:14.343920 14630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:14.343935 14630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:14.348968 14630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:43489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:14.350147 14630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:14.351482 14630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:14.360100 14636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:14.360159 14635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:14.360266 14630 server_base.cc:1034] running on GCE node
W20260810 18:05:14.360148 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:14.360958 14630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:14.362180 14630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:14.363344 14630 hybrid_clock.cc:648] HybridClock initialized: now 1786385114363315 us; error 41 us; skew 500 ppm
I20260810 18:05:14.365270 14630 webserver.cc:460] Webserver started at http://127.28.83.126:42987/ using document root <none> and password file <none>
I20260810 18:05:14.365862 14630 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:14.365924 14630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:14.366168 14630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:14.368198 14630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "68ee238e502a426093f6dae959995783"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "010b897e2591d6024e8e84864ba7a7c9"
server_key_iv: "3f15a3e91ea2638a3848319b8855182c"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.368777 14630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "68ee238e502a426093f6dae959995783"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "010b897e2591d6024e8e84864ba7a7c9"
server_key_iv: "3f15a3e91ea2638a3848319b8855182c"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.372643 14630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:05:14.375172 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:14.376494 14630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:05:14.376637 14630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "68ee238e502a426093f6dae959995783"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "010b897e2591d6024e8e84864ba7a7c9"
server_key_iv: "3f15a3e91ea2638a3848319b8855182c"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.376756 14630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:14.388084 14630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:14.388835 14630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:14.389055 14630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:14.396302 14700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:43489 every 8 connection(s)
I20260810 18:05:14.396330 14630 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:43489
I20260810 18:05:14.397396 14630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:14.400843 14701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:14.403563 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 14630
I20260810 18:05:14.403704 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:14.404027 29005 external_mini_cluster.cc:1428] Setting key 2b21a3540fbbfc2864a4aeac618d8de3
I20260810 18:05:14.407284 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783: Bootstrap starting.
I20260810 18:05:14.409631 14701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:14.410351 14701 log.cc:824] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:14.412588 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783: No bootstrap required, opened a new log
I20260810 18:05:14.413889 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.405559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.28.83.126:43489)
0810 18:05:14.406208 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:14.406223 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:14.406259 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:05:14.406906 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:14.406922 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:14.407322 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:14.407720 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:14.408171 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.409154 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.410162 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.410712 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.410904 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.411735 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:14.411743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:14.411764 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:14.412281 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:14.412357 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:14.412383 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:14.412491 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:14.412618 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 18:05:14.412900 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
I20260810 18:05:14.416867 14701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ee238e502a426093f6dae959995783" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43489 } }
I20260810 18:05:14.417083 14701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:14.417116 14701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ee238e502a426093f6dae959995783, State: Initialized, Role: FOLLOWER
I20260810 18:05:14.417538 14701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ee238e502a426093f6dae959995783" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43489 } }
I20260810 18:05:14.417673 14701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:14.417719 14701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:14.417791 14701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:14.418704 14701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ee238e502a426093f6dae959995783" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43489 } }
I20260810 18:05:14.419080 14701 leader_election.cc:302] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68ee238e502a426093f6dae959995783; no voters:
I20260810 18:05:14.419450 14701 leader_election.cc:288] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:14.419595 14706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:14.419824 14706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [term 1 LEADER]: Becoming Leader. State: Replica: 68ee238e502a426093f6dae959995783, State: Running, Role: LEADER
I20260810 18:05:14.420113 14706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ee238e502a426093f6dae959995783" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43489 } }
I20260810 18:05:14.420694 14701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:14.421851 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68ee238e502a426093f6dae959995783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ee238e502a426093f6dae959995783" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43489 } } }
I20260810 18:05:14.421993 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:14.422267 14708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68ee238e502a426093f6dae959995783. Latest consensus state: current_term: 1 leader_uuid: "68ee238e502a426093f6dae959995783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ee238e502a426093f6dae959995783" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43489 } } }
I20260810 18:05:14.422359 14708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:14.422395 14715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:14.425231 14715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:14.430477 14715 catalog_manager.cc:1347] Generated new cluster ID: 795e0cc66fbe46e0ad37db38073ec80f
I20260810 18:05:14.430557 14715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:14.437826 14715 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:14.439043 14715 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:14.448899 14715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783: Generated new TSK 0
I20260810 18:05:14.449827 14715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:14.452459 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43489
--builtin_ntp_servers=127.28.83.84:33191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:05:14.568245 14725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:14.568593 14725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:14.568637 14725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:14.572930 14725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:14.573002 14725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:14.573091 14725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:05:14.578053 14725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:14.579275 14725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:14.580873 14725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:05:14.590996 14725 server_base.cc:1034] running on GCE node
W20260810 18:05:14.590953 14734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:14.590953 14730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:14.590957 14731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:14.591710 14725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:14.592427 14725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:14.593592 14725 hybrid_clock.cc:648] HybridClock initialized: now 1786385114593575 us; error 29 us; skew 500 ppm
I20260810 18:05:14.595660 14725 webserver.cc:460] Webserver started at http://127.28.83.65:36207/ using document root <none> and password file <none>
I20260810 18:05:14.596290 14725 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:14.596359 14725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:14.596589 14725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:14.598868 14725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "3c2088520aeb4f048b196a3d8d79c256"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "879aada84a61e3d574831815097f16f4"
server_key_iv: "ba88d447bbfdae371310b7ce2e5db7fe"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.599462 14725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "3c2088520aeb4f048b196a3d8d79c256"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "879aada84a61e3d574831815097f16f4"
server_key_iv: "ba88d447bbfdae371310b7ce2e5db7fe"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.603320 14725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:14.605983 14741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:14.607306 14725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:14.607455 14725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "3c2088520aeb4f048b196a3d8d79c256"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "879aada84a61e3d574831815097f16f4"
server_key_iv: "ba88d447bbfdae371310b7ce2e5db7fe"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.607587 14725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:14.620618 14725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:14.621551 14725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:14.621783 14725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:14.622525 14725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:14.623615 14725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:14.623687 14725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:14.623772 14725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:14.623816 14725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:14.633590 14725 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:34869
I20260810 18:05:14.633610 14860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:34869 every 8 connection(s)
I20260810 18:05:14.634577 14725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:05:14.639853 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 14725
I20260810 18:05:14.640004 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:05:14.640297 29005 external_mini_cluster.cc:1428] Setting key adb08782604bc9ff5ea9323f23553cde
I20260810 18:05:14.642335 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43489
--builtin_ntp_servers=127.28.83.84:33191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:05:14.644644 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.636526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:41181 (local address 127.28.83.126:43489)
0810 18:05:14.636715 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:14.636719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:14.637790 (+ 1071us) server_negotiation.cc:407] Connection header received
0810 18:05:14.639229 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:14.639231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:14.639281 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:14.639361 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:14.640478 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.641243 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.642494 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.642765 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.642889 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.644110 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:14.644114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:14.644116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:14.644131 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:14.644159 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:14.644168 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:14.644252 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:14.644420 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 18:05:14.644487 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 18:05:14.645881 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.636923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43489 (local address 127.28.83.65:41181)
0810 18:05:14.637610 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:14.637663 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:14.639019 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:14.639534 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:14.639541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:14.639636 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:14.640243 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:14.640261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.641392 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.641396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:14.642349 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:14.642360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.642935 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.642939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:14.643127 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.643812 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:05:14.643814 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:14.643856 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:05:14.643927 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:14.644254 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:14.644267 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:14.644276 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:05:14.644419 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 18:05:14.644727 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
I20260810 18:05:14.647279 14861 heartbeater.cc:347] Connected to a master server at 127.28.83.126:43489
I20260810 18:05:14.647550 14861 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:14.648118 14861 heartbeater.cc:511] Master 127.28.83.126:43489 requested a full tablet report, sending...
I20260810 18:05:14.649986 14663 ts_manager.cc:194] Registered new tserver with Master: 3c2088520aeb4f048b196a3d8d79c256 (127.28.83.65:34869)
I20260810 18:05:14.651836 14663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.65:41181
W20260810 18:05:14.760646 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:14.760938 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:14.760980 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:14.765045 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:14.765115 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:14.765170 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:05:14.770162 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:14.771260 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:14.772739 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:14.782662 14873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:14.782662 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:14.782684 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:14.782876 14864 server_base.cc:1034] running on GCE node
I20260810 18:05:14.783512 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:14.784220 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:14.785480 14864 hybrid_clock.cc:648] HybridClock initialized: now 1786385114785432 us; error 51 us; skew 500 ppm
I20260810 18:05:14.787549 14864 webserver.cc:460] Webserver started at http://127.28.83.66:37253/ using document root <none> and password file <none>
I20260810 18:05:14.788182 14864 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:14.788254 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:14.788532 14864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:14.790776 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "d16dc90cf54449faa106fba86e795130"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "78004b18849b5ae3eb47839e8d9f9a4e"
server_key_iv: "d7baeb0136f8c3895aad91535ad11dfa"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.791394 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "d16dc90cf54449faa106fba86e795130"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "78004b18849b5ae3eb47839e8d9f9a4e"
server_key_iv: "d7baeb0136f8c3895aad91535ad11dfa"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.795367 14864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:14.798097 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:14.799508 14864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:14.799680 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "d16dc90cf54449faa106fba86e795130"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "78004b18849b5ae3eb47839e8d9f9a4e"
server_key_iv: "d7baeb0136f8c3895aad91535ad11dfa"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.799814 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:14.813958 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:14.814759 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:14.814977 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:14.815614 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:14.816602 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:14.816651 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:14.816731 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:14.816768 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:14.826568 14864 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:33675
I20260810 18:05:14.826599 14996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:33675 every 8 connection(s)
I20260810 18:05:14.827576 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:05:14.830412 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 14864
I20260810 18:05:14.830547 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:05:14.830897 29005 external_mini_cluster.cc:1428] Setting key 522a6132aeb170c9c16da9b4a7b5b064
I20260810 18:05:14.833081 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43489
--builtin_ntp_servers=127.28.83.84:33191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:05:14.838088 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.829371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:60613 (local address 127.28.83.126:43489)
0810 18:05:14.829577 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:14.829583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:14.830853 (+ 1270us) server_negotiation.cc:407] Connection header received
0810 18:05:14.832575 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:14.832578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:14.832624 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:14.832693 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:14.833970 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.834490 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.835782 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.836040 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.836132 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.837553 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:14.837557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:14.837575 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:14.837590 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:14.837609 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:14.837618 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:14.837681 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:14.837907 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 18:05:14.837965 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 18:05:14.839411 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:14.829841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43489 (local address 127.28.83.66:60613)
0810 18:05:14.830656 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:14.830719 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:14.832334 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:14.832915 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:14.832926 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:14.833040 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:14.833817 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:14.833840 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.834644 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.834651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:14.835653 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:14.835663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:14.836224 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:14.836231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:14.836481 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:14.837186 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:05:14.837190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:14.837251 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:05:14.837343 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:14.837714 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:14.837727 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:14.837736 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:05:14.837887 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 18:05:14.838233 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
I20260810 18:05:14.840740 14997 heartbeater.cc:347] Connected to a master server at 127.28.83.126:43489
I20260810 18:05:14.840991 14997 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:14.841615 14997 heartbeater.cc:511] Master 127.28.83.126:43489 requested a full tablet report, sending...
I20260810 18:05:14.842684 14663 ts_manager.cc:194] Registered new tserver with Master: d16dc90cf54449faa106fba86e795130 (127.28.83.66:33675)
I20260810 18:05:14.843271 14663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.66:60613
W20260810 18:05:14.951615 15000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:14.951943 15000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:14.952031 15000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:14.956246 15000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:14.956354 15000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:14.956447 15000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:05:14.961529 15000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:43489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:14.962709 15000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:14.964162 15000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:14.974097 15009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:14.974282 15000 server_base.cc:1034] running on GCE node
W20260810 18:05:14.974145 15005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:14.974099 15006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:14.974887 15000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:14.975570 15000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:14.976779 15000 hybrid_clock.cc:648] HybridClock initialized: now 1786385114976767 us; error 31 us; skew 500 ppm
I20260810 18:05:14.978816 15000 webserver.cc:460] Webserver started at http://127.28.83.67:46683/ using document root <none> and password file <none>
I20260810 18:05:14.979429 15000 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:14.979532 15000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:14.979781 15000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:14.982098 15000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "83ffe5e640c4dc01f928e7854a9307eb"
server_key_iv: "f13c89c4c964533865c80feab708706e"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.982708 15000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "83ffe5e640c4dc01f928e7854a9307eb"
server_key_iv: "f13c89c4c964533865c80feab708706e"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.986548 15000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 18:05:14.989038 15015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:14.990375 15000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:05:14.990520 15000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db"
format_stamp: "Formatted at 2026-08-10 18:05:14 on dist-test-slave-srgv"
server_key: "83ffe5e640c4dc01f928e7854a9307eb"
server_key_iv: "f13c89c4c964533865c80feab708706e"
server_key_version: "encryptionkey@0"
I20260810 18:05:14.990649 15000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:15.002565 15000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:15.003402 15000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:15.003604 15000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:15.004268 15000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:15.005280 15000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:15.005352 15000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:15.005484 15000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:15.005532 15000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:15.015054 15000 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:45843
I20260810 18:05:15.015075 15131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:45843 every 8 connection(s)
I20260810 18:05:15.016021 15000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:05:15.020860 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 15000
I20260810 18:05:15.020998 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:05:15.021351 29005 external_mini_cluster.cc:1428] Setting key a9d5cfcc6aeef62bd302cdaf60b92dc1
I20260810 18:05:15.026247 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.018148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:53831 (local address 127.28.83.126:43489)
0810 18:05:15.018314 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:15.018319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:15.019506 (+ 1187us) server_negotiation.cc:407] Connection header received
0810 18:05:15.021010 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:15.021014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:15.021079 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:15.021199 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:15.022408 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.023015 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.024317 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.024536 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.024618 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.025740 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:15.025743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:15.025745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:15.025758 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:15.025776 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:15.025784 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:15.025847 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:15.026060 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 18:05:15.026111 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:05:15.027148 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.018645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43489 (local address 127.28.83.67:53831)
0810 18:05:15.019346 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:15.019386 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:15.020756 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:15.021444 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:15.021454 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:15.021565 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:15.022277 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.022298 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.023171 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.023175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.024199 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.024209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.024679 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.024682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.024828 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.025456 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:05:15.025459 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:15.025519 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:05:15.025588 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:15.025893 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:15.025902 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:15.025907 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:05:15.026048 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 18:05:15.026337 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":151,"threads_started":1}
I20260810 18:05:15.028333 15132 heartbeater.cc:347] Connected to a master server at 127.28.83.126:43489
I20260810 18:05:15.028585 15132 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:15.029120 15132 heartbeater.cc:511] Master 127.28.83.126:43489 requested a full tablet report, sending...
I20260810 18:05:15.030220 14663 ts_manager.cc:194] Registered new tserver with Master: 5d7bdb0ad9e44f4e9c5d26f5c14af8db (127.28.83.67:45843)
I20260810 18:05:15.030824 14663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.67:53831
I20260810 18:05:15.035557 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:05:15.041476 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.037530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.28.83.126:43489)
0810 18:05:15.037669 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:15.037675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:15.037862 (+ 187us) server_negotiation.cc:407] Connection header received
0810 18:05:15.038010 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:15.038014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:15.038059 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:15.038152 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:15.038443 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.038965 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.039829 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.040063 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.040174 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.041009 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:15.041012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:15.041015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:15.041028 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:15.041057 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:15.041066 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:15.041145 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:15.041239 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 18:05:15.041299 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:05:15.044458 14663 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:05:15.047699 14663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:05:15.062659 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.056568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.28.83.67:45843)
0810 18:05:15.057729 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:15.057745 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:15.057773 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:05:15.057858 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:15.057863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:15.058086 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:15.058233 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:15.058647 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.059559 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.061202 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.061940 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.062079 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.062264 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:15.062377 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 18:05:15.062474 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":129,"threads_started":1}
I20260810 18:05:15.062654 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.056386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:45843 (local address 127.0.0.1:55898)
0810 18:05:15.056883 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:15.056931 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:15.057067 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:15.058268 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:15.058276 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:15.058299 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:15.058505 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.058511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.059663 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.059669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.061058 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.061069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.062073 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.062077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.062188 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.062226 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 18:05:15.062346 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 18:05:15.062474 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":14,"thread_start_us":99,"threads_started":1}
I20260810 18:05:15.063004 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.057466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.28.83.65:34869)
0810 18:05:15.057884 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:15.057897 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:15.057924 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:05:15.058004 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:15.058008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:15.058143 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:15.058322 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:15.058655 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.059818 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.061234 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.061965 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.062066 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.062266 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:15.062726 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 18:05:15.062846 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":127,"threads_started":1}
I20260810 18:05:15.063185 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.057134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:34869 (local address 127.0.0.1:57050)
0810 18:05:15.057774 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:15.057789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:15.057877 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:15.058313 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:15.058317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:15.058335 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:15.058505 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.058513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.059949 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.059953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.061102 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.061112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.062338 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.062341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.062432 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.062462 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:05:15.062531 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:05:15.062601 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":66,"thread_start_us":100,"threads_started":1}
I20260810 18:05:15.063747 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.056590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:33675 (local address 127.0.0.1:55668)
0810 18:05:15.057629 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:15.057650 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:15.057772 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:15.059055 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:15.059058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:15.059076 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:15.059175 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.059180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.060497 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.060500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.061546 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.061566 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.063337 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.063342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.063453 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.063475 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:05:15.063543 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:05:15.063609 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260810 18:05:15.064308 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.057142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.28.83.66:33675)
0810 18:05:15.058616 (+ 1474us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:15.058631 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:15.058654 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:05:15.058727 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:15.058731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:15.058904 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:15.059053 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:15.059307 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.060373 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.061688 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.062438 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.062545 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.062708 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:15.064012 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0810 18:05:15.064141 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1334,"thread_start_us":137,"threads_started":1}
I20260810 18:05:15.066126 15065 tablet_service.cc:1467] Processing CreateTablet for tablet 28eecc4a07064602923c470708c217b7 (DEFAULT_TABLE table=test-table [id=38e003e7ca0f46c29fe6ba562fdb273a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:15.066130 14794 tablet_service.cc:1467] Processing CreateTablet for tablet 28eecc4a07064602923c470708c217b7 (DEFAULT_TABLE table=test-table [id=38e003e7ca0f46c29fe6ba562fdb273a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:15.067272 14794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eecc4a07064602923c470708c217b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:15.067273 15065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eecc4a07064602923c470708c217b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:15.067734 14929 tablet_service.cc:1467] Processing CreateTablet for tablet 28eecc4a07064602923c470708c217b7 (DEFAULT_TABLE table=test-table [id=38e003e7ca0f46c29fe6ba562fdb273a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:15.068679 14929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eecc4a07064602923c470708c217b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:15.074716 15149 tablet_bootstrap.cc:492] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: Bootstrap starting.
I20260810 18:05:15.076082 15151 tablet_bootstrap.cc:492] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256: Bootstrap starting.
I20260810 18:05:15.077008 15149 tablet_bootstrap.cc:654] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:15.077785 15150 tablet_bootstrap.cc:492] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: Bootstrap starting.
I20260810 18:05:15.077914 15149 log.cc:824] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:15.078056 15151 tablet_bootstrap.cc:654] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:15.078893 15151 log.cc:824] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:15.080075 15150 tablet_bootstrap.cc:654] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:15.080358 15149 tablet_bootstrap.cc:492] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: No bootstrap required, opened a new log
I20260810 18:05:15.080565 15149 ts_tablet_manager.cc:1397] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:05:15.080883 15150 log.cc:824] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:15.081126 15151 tablet_bootstrap.cc:492] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256: No bootstrap required, opened a new log
I20260810 18:05:15.081336 15151 ts_tablet_manager.cc:1397] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:05:15.082973 15150 tablet_bootstrap.cc:492] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: No bootstrap required, opened a new log
I20260810 18:05:15.083179 15150 ts_tablet_manager.cc:1397] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:05:15.084338 15149 raft_consensus.cc:348] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } }
I20260810 18:05:15.084591 15149 raft_consensus.cc:374] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:15.084645 15149 raft_consensus.cc:729] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d16dc90cf54449faa106fba86e795130, State: Initialized, Role: FOLLOWER
I20260810 18:05:15.085158 15149 consensus_queue.cc:260] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } }
I20260810 18:05:15.085845 15151 raft_consensus.cc:348] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } }
I20260810 18:05:15.086113 15151 raft_consensus.cc:374] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:15.086177 15151 raft_consensus.cc:729] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2088520aeb4f048b196a3d8d79c256, State: Initialized, Role: FOLLOWER
I20260810 18:05:15.086313 15149 ts_tablet_manager.cc:1428] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:05:15.086721 14997 heartbeater.cc:503] Master 127.28.83.126:43489 was elected leader, sending a full tablet report...
I20260810 18:05:15.086638 15151 consensus_queue.cc:260] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } }
I20260810 18:05:15.087383 14861 heartbeater.cc:503] Master 127.28.83.126:43489 was elected leader, sending a full tablet report...
I20260810 18:05:15.087711 15151 ts_tablet_manager.cc:1428] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:05:15.087585 15150 raft_consensus.cc:348] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } }
I20260810 18:05:15.087833 15150 raft_consensus.cc:374] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:15.087888 15150 raft_consensus.cc:729] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d7bdb0ad9e44f4e9c5d26f5c14af8db, State: Initialized, Role: FOLLOWER
I20260810 18:05:15.088327 15150 consensus_queue.cc:260] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } }
I20260810 18:05:15.088974 15132 heartbeater.cc:503] Master 127.28.83.126:43489 was elected leader, sending a full tablet report...
I20260810 18:05:15.089320 15150 ts_tablet_manager.cc:1428] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 18:05:15.135888 14862 tablet.cc:2369] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:05:15.208371 15156 raft_consensus.cc:482] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:15.208618 15156 raft_consensus.cc:504] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } }
I20260810 18:05:15.209829 15156 leader_election.cc:288] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d7bdb0ad9e44f4e9c5d26f5c14af8db (127.28.83.67:45843), d16dc90cf54449faa106fba86e795130 (127.28.83.66:33675)
I20260810 18:05:15.215392 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.210229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:51841 (local address 127.28.83.67:45843)
0810 18:05:15.210374 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:15.210381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:15.210621 (+ 240us) server_negotiation.cc:407] Connection header received
0810 18:05:15.210840 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:15.210845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:15.210897 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:15.210969 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:15.211228 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.212007 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.213986 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.214607 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.214744 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.214796 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:15.215167 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 18:05:15.215251 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:05:15.215392 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.210229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:60183 (local address 127.28.83.66:33675)
0810 18:05:15.210371 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:15.210376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:15.210551 (+ 175us) server_negotiation.cc:407] Connection header received
0810 18:05:15.210717 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:15.210722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:15.210801 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:15.210878 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:15.211217 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.212013 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.213986 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.214611 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.214744 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.214796 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:15.215167 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 18:05:15.215252 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:05:15.215437 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.210084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:45843 (local address 127.28.83.65:51841)
0810 18:05:15.210519 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:15.210531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:15.210665 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:15.210985 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:15.210987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:15.210999 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:15.211113 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.211122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.212122 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.212126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.213814 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.213826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.214744 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.214748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.214945 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.215025 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 18:05:15.215107 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:05:15.215247 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":363,"thread_start_us":83,"threads_started":1}
I20260810 18:05:15.215447 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.210084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:33675 (local address 127.28.83.65:60183)
0810 18:05:15.210464 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:15.210476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:15.210605 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:15.210897 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:15.210899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:15.210913 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:15.211072 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.211081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.212122 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.212126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.213814 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:15.213826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.214744 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.214748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:15.214948 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.215025 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 18:05:15.215107 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:05:15.215247 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":271,"thread_start_us":91,"threads_started":1}
I20260810 18:05:15.216297 15086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eecc4a07064602923c470708c217b7" candidate_uuid: "3c2088520aeb4f048b196a3d8d79c256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" is_pre_election: true
I20260810 18:05:15.216391 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eecc4a07064602923c470708c217b7" candidate_uuid: "3c2088520aeb4f048b196a3d8d79c256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16dc90cf54449faa106fba86e795130" is_pre_election: true
I20260810 18:05:15.216648 15086 raft_consensus.cc:2454] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c2088520aeb4f048b196a3d8d79c256 in term 0.
I20260810 18:05:15.216711 14951 raft_consensus.cc:2454] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c2088520aeb4f048b196a3d8d79c256 in term 0.
I20260810 18:05:15.217190 14744 leader_election.cc:302] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c2088520aeb4f048b196a3d8d79c256, 5d7bdb0ad9e44f4e9c5d26f5c14af8db, d16dc90cf54449faa106fba86e795130; no voters:
I20260810 18:05:15.217497 15156 raft_consensus.cc:2781] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:05:15.217586 15156 raft_consensus.cc:482] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:05:15.217629 15156 raft_consensus.cc:3037] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:15.218744 15156 raft_consensus.cc:504] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } }
I20260810 18:05:15.219110 15156 leader_election.cc:288] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [CANDIDATE]: Term 1 election: Requested vote from peers 5d7bdb0ad9e44f4e9c5d26f5c14af8db (127.28.83.67:45843), d16dc90cf54449faa106fba86e795130 (127.28.83.66:33675)
I20260810 18:05:15.219424 15086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eecc4a07064602923c470708c217b7" candidate_uuid: "3c2088520aeb4f048b196a3d8d79c256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db"
I20260810 18:05:15.219498 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eecc4a07064602923c470708c217b7" candidate_uuid: "3c2088520aeb4f048b196a3d8d79c256" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16dc90cf54449faa106fba86e795130"
I20260810 18:05:15.219558 15086 raft_consensus.cc:3037] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:15.219641 14951 raft_consensus.cc:3037] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:15.220667 14951 raft_consensus.cc:2454] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c2088520aeb4f048b196a3d8d79c256 in term 1.
I20260810 18:05:15.220827 15086 raft_consensus.cc:2454] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c2088520aeb4f048b196a3d8d79c256 in term 1.
I20260810 18:05:15.220994 14742 leader_election.cc:302] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2088520aeb4f048b196a3d8d79c256, d16dc90cf54449faa106fba86e795130; no voters:
I20260810 18:05:15.221216 15156 raft_consensus.cc:2781] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:15.221525 15156 raft_consensus.cc:686] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [term 1 LEADER]: Becoming Leader. State: Replica: 3c2088520aeb4f048b196a3d8d79c256, State: Running, Role: LEADER
I20260810 18:05:15.221930 15156 consensus_queue.cc:237] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } }
I20260810 18:05:15.224862 14663 catalog_manager.cc:5495] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c2088520aeb4f048b196a3d8d79c256 (127.28.83.65). New cstate: current_term: 1 leader_uuid: "3c2088520aeb4f048b196a3d8d79c256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c2088520aeb4f048b196a3d8d79c256" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 34869 } health_report { overall_health: HEALTHY } } }
I20260810 18:05:15.253314 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.249765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.28.83.65:34869)
0810 18:05:15.249914 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:15.249920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:15.249939 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:05:15.250033 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:15.250035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:15.250092 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:15.250209 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:15.250469 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.250981 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.251829 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:15.252048 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:15.252159 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:15.252482 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:15.252906 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:15.253040 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:15.253099 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 18:05:15.253153 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:05:15.259189 15086 raft_consensus.cc:1261] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [term 1 FOLLOWER]: Refusing update from remote peer 3c2088520aeb4f048b196a3d8d79c256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:15.259189 14951 raft_consensus.cc:1261] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [term 1 FOLLOWER]: Refusing update from remote peer 3c2088520aeb4f048b196a3d8d79c256: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:15.259934 15160 consensus_queue.cc:1035] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d7bdb0ad9e44f4e9c5d26f5c14af8db" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 45843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:15.260174 15156 consensus_queue.cc:1035] T 28eecc4a07064602923c470708c217b7 P 3c2088520aeb4f048b196a3d8d79c256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d16dc90cf54449faa106fba86e795130" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 33675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:05:15.266963 15133 tablet.cc:2369] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:05:15.267689 15166 mvcc.cc:205] Tried to move back new op lower bound from 7317033432094498816 to 7317033431952363520. Current Snapshot: MvccSnapshot[applied={T|T < 7317033432094498816}]
I20260810 18:05:15.277937 14662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:15.278160 14662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:15.280954 14662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68ee238e502a426093f6dae959995783: Sending DeleteTablet for 3 replicas of tablet 28eecc4a07064602923c470708c217b7
I20260810 18:05:15.281828 14929 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eecc4a07064602923c470708c217b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:15 UTC) from {username='slave'} at 127.0.0.1:55668
I20260810 18:05:15.281812 15065 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eecc4a07064602923c470708c217b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:15 UTC) from {username='slave'} at 127.0.0.1:55898
I20260810 18:05:15.282459 15175 tablet_replica.cc:330] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: stopping tablet replica
I20260810 18:05:15.282805 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eecc4a07064602923c470708c217b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:15 UTC) from {username='slave'} at 127.0.0.1:57050
I20260810 18:05:15.283210 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 14725
I20260810 18:05:15.285503 15175 raft_consensus.cc:2229] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:15.285856 15175 raft_consensus.cc:2258] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:15.286309 15174 tablet_replica.cc:330] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: stopping tablet replica
I20260810 18:05:15.286629 15174 raft_consensus.cc:2229] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:15.286921 15174 raft_consensus.cc:2258] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:15.287717 15175 ts_tablet_manager.cc:1905] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:15.288197 15174 ts_tablet_manager.cc:1905] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:15.290537 15174 ts_tablet_manager.cc:1918] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:15.290661 15174 log.cc:1196] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/28eecc4a07064602923c470708c217b7
I20260810 18:05:15.290658 15175 ts_tablet_manager.cc:1918] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:15.290755 15175 log.cc:1196] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/28eecc4a07064602923c470708c217b7
I20260810 18:05:15.291076 15175 ts_tablet_manager.cc:1939] T 28eecc4a07064602923c470708c217b7 P 5d7bdb0ad9e44f4e9c5d26f5c14af8db: Deleting consensus metadata
I20260810 18:05:15.291062 15174 ts_tablet_manager.cc:1939] T 28eecc4a07064602923c470708c217b7 P d16dc90cf54449faa106fba86e795130: Deleting consensus metadata
W20260810 18:05:15.291508 14649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:34869 (error 108)
I20260810 18:05:15.292348 14649 catalog_manager.cc:4841] TS 5d7bdb0ad9e44f4e9c5d26f5c14af8db (127.28.83.67:45843): tablet 28eecc4a07064602923c470708c217b7 (table test-table [id=38e003e7ca0f46c29fe6ba562fdb273a]) successfully deleted
I20260810 18:05:15.292348 14647 catalog_manager.cc:4841] TS d16dc90cf54449faa106fba86e795130 (127.28.83.66:33675): tablet 28eecc4a07064602923c470708c217b7 (table test-table [id=38e003e7ca0f46c29fe6ba562fdb273a]) successfully deleted
I20260810 18:05:15.292886 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 14864
I20260810 18:05:15.293186 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:15.292881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:34869 (local address 127.0.0.1:57064)
0810 18:05:15.293014 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:15.293090 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 18:05:15.295356 14649 catalog_manager.cc:4584] TS 3c2088520aeb4f048b196a3d8d79c256 (127.28.83.65:34869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28eecc4a07064602923c470708c217b7: Network error: Client connection negotiation failed: client connection to 127.28.83.65:34869: connect: Connection refused (error 111)
I20260810 18:05:15.300339 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 15000
I20260810 18:05:15.307206 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 14630
2026-08-10T18:05:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1111 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:15 dist-test-slave-srgv krb5kdc[15181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:15 dist-test-slave-srgv krb5kdc[15181](info): set up 2 sockets
Aug 10 18:05:15 dist-test-slave-srgv krb5kdc[15181](info): commencing operation
krb5kdc: starting...
W20260810 18:05:16.388039 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:16 dist-test-slave-srgv krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:16.517066 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:42171
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:16.631100 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:16.631470 15195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:16.631563 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:16.636274 15195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:16.636425 15195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:16.636466 15195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:16.636513 15195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:16.636551 15195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:16.642570 15195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:42171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:42171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:16.643831 15195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:16.644958 15195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:16.654201 15200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:16.654217 15201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:16.654446 15195 server_base.cc:1034] running on GCE node
W20260810 18:05:16.654701 15203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:16.655398 15195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:16.656633 15195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:16.657807 15195 hybrid_clock.cc:648] HybridClock initialized: now 1786385116657781 us; error 47 us; skew 500 ppm
Aug 10 18:05:16 dist-test-slave-srgv krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385116, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:16.660818 15195 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:16.661875 15195 webserver.cc:460] Webserver started at http://127.28.83.126:41051/ using document root <none> and password file <none>
I20260810 18:05:16.662492 15195 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:16.662590 15195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:16.662840 15195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:16.665153 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "17e09747a4b3409f9a223438ef047d77"
format_stamp: "Formatted at 2026-08-10 18:05:16 on dist-test-slave-srgv"
server_key: "26132d1e6b0038c8d1f0feba707f03ea"
server_key_iv: "af6a0b2da92dfc046fa1b89446a5c830"
server_key_version: "encryptionkey@0"
I20260810 18:05:16.665853 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "17e09747a4b3409f9a223438ef047d77"
format_stamp: "Formatted at 2026-08-10 18:05:16 on dist-test-slave-srgv"
server_key: "26132d1e6b0038c8d1f0feba707f03ea"
server_key_iv: "af6a0b2da92dfc046fa1b89446a5c830"
server_key_version: "encryptionkey@0"
I20260810 18:05:16.669976 15195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:05:16.672551 15211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:16.673956 15195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:16.674122 15195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "17e09747a4b3409f9a223438ef047d77"
format_stamp: "Formatted at 2026-08-10 18:05:16 on dist-test-slave-srgv"
server_key: "26132d1e6b0038c8d1f0feba707f03ea"
server_key_iv: "af6a0b2da92dfc046fa1b89446a5c830"
server_key_version: "encryptionkey@0"
I20260810 18:05:16.674260 15195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:16.685089 15195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:16.688462 15195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:16.688709 15195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:16.696375 15195 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:42171
I20260810 18:05:16.696449 15263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:42171 every 8 connection(s)
I20260810 18:05:16.697499 15195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:16.700404 15264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:16.704298 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 15195
I20260810 18:05:16.704469 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:16.704780 29005 external_mini_cluster.cc:1428] Setting key 0c390734412a12e2fbdad4905a5529c0
I20260810 18:05:16.706568 15264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77: Bootstrap starting.
I20260810 18:05:16.709205 15264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:16.710078 15264 log.cc:824] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:16.712519 15264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77: No bootstrap required, opened a new log
Aug 10 18:05:16 dist-test-slave-srgv krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:16.717666 15264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e09747a4b3409f9a223438ef047d77" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42171 } }
I20260810 18:05:16.717971 15264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:16.718055 15264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e09747a4b3409f9a223438ef047d77, State: Initialized, Role: FOLLOWER
I20260810 18:05:16.718586 15264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e09747a4b3409f9a223438ef047d77" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42171 } }
I20260810 18:05:16.718807 15264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:16.718883 15264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:16.718981 15264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:16.720266 15264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e09747a4b3409f9a223438ef047d77" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42171 } }
I20260810 18:05:16.720655 15264 leader_election.cc:302] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17e09747a4b3409f9a223438ef047d77; no voters:
I20260810 18:05:16.721010 15264 leader_election.cc:288] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:16.721038 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:16.706255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.28.83.126:42171)
0810 18:05:16.706696 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:16.706739 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:16.706790 (+ 51us) server_negotiation.cc:407] Connection header received
0810 18:05:16.707486 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:16.707503 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:16.707843 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:16.708134 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:16.709118 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:16.710299 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:16.711441 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:16.711928 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:16.712066 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:16.715171 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:16.715178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:16.715199 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:16.715237 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:16.717449 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:16.717938 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:16.717945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:16.717951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:16.718150 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:16.718479 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:16.718482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:16.718484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:16.719140 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:16.719360 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:16.719593 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 18:05:16.719936 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":148,"threads_started":1}
I20260810 18:05:16.721154 15269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:16.721398 15269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [term 1 LEADER]: Becoming Leader. State: Replica: 17e09747a4b3409f9a223438ef047d77, State: Running, Role: LEADER
I20260810 18:05:16.721778 15269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e09747a4b3409f9a223438ef047d77" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42171 } }
I20260810 18:05:16.722368 15264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:16.724288 15270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17e09747a4b3409f9a223438ef047d77. Latest consensus state: current_term: 1 leader_uuid: "17e09747a4b3409f9a223438ef047d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e09747a4b3409f9a223438ef047d77" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42171 } } }
I20260810 18:05:16.724427 15270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:16.724699 15271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17e09747a4b3409f9a223438ef047d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e09747a4b3409f9a223438ef047d77" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 42171 } } }
I20260810 18:05:16.724936 15271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77 [sys.catalog]: This master's current role is: LEADER
W20260810 18:05:16.727360 15283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:16.727438 15283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:05:16.727528 15284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:16.728338 15284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:16.734027 15284 catalog_manager.cc:1347] Generated new cluster ID: 62b0d65f13f3410ba05c8635a92b34c6
I20260810 18:05:16.734123 15284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:16.740507 15284 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:16.741564 15284 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:16.748880 15284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77: Generated new TSK 0
I20260810 18:05:16.749768 15284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:05:16.789196 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42171
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:16.906075 15291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:16.906380 15291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:16.906476 15291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:16.910662 15291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:16.910734 15291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:16.910847 15291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:05:16.916361 15291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:16.917582 15291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:16.918684 15291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:16.928985 15296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:16.929126 15291 server_base.cc:1034] running on GCE node
W20260810 18:05:16.928992 15297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:16.928978 15299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:16.929929 15291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:16.930644 15291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:16.931835 15291 hybrid_clock.cc:648] HybridClock initialized: now 1786385116931822 us; error 30 us; skew 500 ppm
Aug 10 18:05:16 dist-test-slave-srgv krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385116, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:16.939584 15291 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:05:16.940570 15291 webserver.cc:460] Webserver started at http://127.28.83.65:43711/ using document root <none> and password file <none>
I20260810 18:05:16.941159 15291 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:16.941226 15291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:16.941509 15291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:16.943753 15291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "7df70cd944c347d4b4d9fe81c2d196db"
format_stamp: "Formatted at 2026-08-10 18:05:16 on dist-test-slave-srgv"
server_key: "b04f0d8c38fd72bcc671ea04dcf9f1d6"
server_key_iv: "e77058b772d9d1b359c1996c653ad271"
server_key_version: "encryptionkey@0"
I20260810 18:05:16.944356 15291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "7df70cd944c347d4b4d9fe81c2d196db"
format_stamp: "Formatted at 2026-08-10 18:05:16 on dist-test-slave-srgv"
server_key: "b04f0d8c38fd72bcc671ea04dcf9f1d6"
server_key_iv: "e77058b772d9d1b359c1996c653ad271"
server_key_version: "encryptionkey@0"
I20260810 18:05:16.948380 15291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 18:05:16.951095 15306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:16.952363 15291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 18:05:16.952492 15291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "7df70cd944c347d4b4d9fe81c2d196db"
format_stamp: "Formatted at 2026-08-10 18:05:16 on dist-test-slave-srgv"
server_key: "b04f0d8c38fd72bcc671ea04dcf9f1d6"
server_key_iv: "e77058b772d9d1b359c1996c653ad271"
server_key_version: "encryptionkey@0"
I20260810 18:05:16.952610 15291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:16.972931 15291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:16.976786 15291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:16.977008 15291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:16.977763 15291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:16.978708 15291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:16.978799 15291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:16.978878 15291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:16.978912 15291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:16.988375 15291 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:42113
I20260810 18:05:16.988395 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:42113 every 8 connection(s)
I20260810 18:05:16.989336 15291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:05:16.996878 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 15291
I20260810 18:05:16.997030 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:05:16.997331 29005 external_mini_cluster.cc:1428] Setting key 9a6527a612d75896ec5bc02ef6d3dbfc
Aug 10 18:05:16 dist-test-slave-srgv krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385116, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:17.004169 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:16.991338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:51357 (local address 127.28.83.126:42171)
0810 18:05:16.991484 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:16.991488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:16.992547 (+ 1059us) server_negotiation.cc:407] Connection header received
0810 18:05:16.993608 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:16.993612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:16.993674 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:16.993761 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:16.995127 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:16.995673 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:16.996867 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:16.997221 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:16.997338 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.000000 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:17.000004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:17.000007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:17.000042 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:17.001747 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:17.002417 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:17.002420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:17.002421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:17.002500 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:17.002909 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:17.002911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:17.002913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:17.003096 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:17.003173 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.003907 (+ 734us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.004024 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:05:17.005600 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:16.991685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42171 (local address 127.28.83.65:51357)
0810 18:05:16.992386 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:16.992431 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:16.993317 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:16.993980 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:16.993986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:16.994425 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:16.994956 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:16.994970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:16.995811 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:16.995820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:16.996731 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:16.996741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:16.997387 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:16.997390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:16.997592 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:16.998227 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:16.998246 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:16.999827 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:17.001875 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:17.001884 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:17.001906 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:17.002306 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:17.002590 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:17.002596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:17.002601 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:17.002812 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:17.003206 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:17.003216 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:17.003598 (+ 382us) client_negotiation.cc:769] Sending connection context
0810 18:05:17.003867 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 18:05:17.004295 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":116,"threads_started":1}
I20260810 18:05:17.006960 15426 heartbeater.cc:347] Connected to a master server at 127.28.83.126:42171
I20260810 18:05:17.007238 15426 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:17.007915 15426 heartbeater.cc:511] Master 127.28.83.126:42171 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
I20260810 18:05:17.009719 15228 ts_manager.cc:194] Registered new tserver with Master: 7df70cd944c347d4b4d9fe81c2d196db (127.28.83.65:42113)
I20260810 18:05:17.011080 15228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:51357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:05:17.029314 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42171
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:17.152380 15431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:17.152664 15431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:17.152712 15431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:17.157061 15431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:17.157142 15431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:17.157207 15431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:05:17.163450 15431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:17.164600 15431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:17.165638 15431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:17.176731 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:17.176731 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:17.176725 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:17.177040 15431 server_base.cc:1034] running on GCE node
I20260810 18:05:17.177711 15431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:17.178481 15431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:17.179688 15431 hybrid_clock.cc:648] HybridClock initialized: now 1786385117179651 us; error 52 us; skew 500 ppm
Aug 10 18:05:17 dist-test-slave-srgv krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385117, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:17.183053 15431 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:05:17.184158 15431 webserver.cc:460] Webserver started at http://127.28.83.66:42533/ using document root <none> and password file <none>
I20260810 18:05:17.184780 15431 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:17.184892 15431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:17.185146 15431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:17.187510 15431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b"
format_stamp: "Formatted at 2026-08-10 18:05:17 on dist-test-slave-srgv"
server_key: "7459a67d72f926a3dc2070aabdd0639d"
server_key_iv: "bb6d6bdb62fa3a3011ae6ae93b6bf311"
server_key_version: "encryptionkey@0"
I20260810 18:05:17.188179 15431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b"
format_stamp: "Formatted at 2026-08-10 18:05:17 on dist-test-slave-srgv"
server_key: "7459a67d72f926a3dc2070aabdd0639d"
server_key_iv: "bb6d6bdb62fa3a3011ae6ae93b6bf311"
server_key_version: "encryptionkey@0"
I20260810 18:05:17.192574 15431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 18:05:17.195492 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:17.196940 15431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:05:17.197142 15431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b"
format_stamp: "Formatted at 2026-08-10 18:05:17 on dist-test-slave-srgv"
server_key: "7459a67d72f926a3dc2070aabdd0639d"
server_key_iv: "bb6d6bdb62fa3a3011ae6ae93b6bf311"
server_key_version: "encryptionkey@0"
I20260810 18:05:17.197288 15431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:17.215929 15431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:17.219765 15431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:17.220003 15431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:17.220671 15431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:17.221647 15431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:17.221721 15431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:17.221797 15431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:17.221855 15431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:17.232493 15431 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:42179
I20260810 18:05:17.232519 15567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:42179 every 8 connection(s)
I20260810 18:05:17.233561 15431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:05:17.237635 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 15431
I20260810 18:05:17.237752 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:05:17.238036 29005 external_mini_cluster.cc:1428] Setting key 5e738c5758d30c89f60a5a8097fa49b7
Aug 10 18:05:17 dist-test-slave-srgv krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385117, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
I20260810 18:05:17.249254 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.235668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:51913 (local address 127.28.83.126:42171)
0810 18:05:17.235836 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.235842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.237097 (+ 1255us) server_negotiation.cc:407] Connection header received
0810 18:05:17.238246 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.238249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.238312 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.238433 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:17.239924 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.240560 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.241872 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.242113 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.242236 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.245044 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:17.245053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:17.245056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:17.245083 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:17.246797 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:17.247433 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:17.247436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:17.247438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:17.247531 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:17.247939 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:17.247944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:17.247946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:17.248214 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:17.248343 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.248910 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.249077 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 18:05:17.250404 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.236169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42171 (local address 127.28.83.66:51913)
0810 18:05:17.236923 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:17.236972 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:17.237977 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:17.238667 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:17.238677 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:17.239146 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:17.239772 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.239788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.240706 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.240709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.241739 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.241750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.242290 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.242294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.242480 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.243184 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:17.243202 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:17.244832 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:17.246950 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:17.246957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:17.246977 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:17.247308 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:17.247637 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:17.247640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:17.247642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:17.247829 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:17.248334 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:17.248342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:17.248648 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 18:05:17.248860 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 18:05:17.249291 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:05:17.251946 15568 heartbeater.cc:347] Connected to a master server at 127.28.83.126:42171
I20260810 18:05:17.252259 15568 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:17.252914 15568 heartbeater.cc:511] Master 127.28.83.126:42171 requested a full tablet report, sending...
I20260810 18:05:17.254321 15228 ts_manager.cc:194] Registered new tserver with Master: 9d08ccefa54a4fe98170d16bf1fe0a1b (127.28.83.66:42179)
I20260810 18:05:17.254983 15228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:51913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:05:17.269939 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42171
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:17.390908 15573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:17.391239 15573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:17.391288 15573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:17.395684 15573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:17.395780 15573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:17.395844 15573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:05:17.401839 15573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:42171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:17.403038 15573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:17.404153 15573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:17.414654 15578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:17.414637 15579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:17.414722 15573 server_base.cc:1034] running on GCE node
W20260810 18:05:17.414641 15581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:17.415367 15573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:17.416482 15573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:17.417698 15573 hybrid_clock.cc:648] HybridClock initialized: now 1786385117417666 us; error 58 us; skew 500 ppm
Aug 10 18:05:17 dist-test-slave-srgv krb5kdc[15181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385117, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:17.420887 15573 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:05:17.422142 15573 webserver.cc:460] Webserver started at http://127.28.83.67:33575/ using document root <none> and password file <none>
I20260810 18:05:17.422785 15573 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:17.422919 15573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:17.423207 15573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:17.425727 15573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "dccce2cdf2a843a8895dd4ad30f4f85c"
format_stamp: "Formatted at 2026-08-10 18:05:17 on dist-test-slave-srgv"
server_key: "7ba601dde6b65605d819f171f3152901"
server_key_iv: "6e3aa733744813cd1d1767d606a3fb7a"
server_key_version: "encryptionkey@0"
I20260810 18:05:17.426411 15573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "dccce2cdf2a843a8895dd4ad30f4f85c"
format_stamp: "Formatted at 2026-08-10 18:05:17 on dist-test-slave-srgv"
server_key: "7ba601dde6b65605d819f171f3152901"
server_key_iv: "6e3aa733744813cd1d1767d606a3fb7a"
server_key_version: "encryptionkey@0"
I20260810 18:05:17.430742 15573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 18:05:17.433596 15588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:17.435081 15573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:17.435292 15573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "dccce2cdf2a843a8895dd4ad30f4f85c"
format_stamp: "Formatted at 2026-08-10 18:05:17 on dist-test-slave-srgv"
server_key: "7ba601dde6b65605d819f171f3152901"
server_key_iv: "6e3aa733744813cd1d1767d606a3fb7a"
server_key_version: "encryptionkey@0"
I20260810 18:05:17.435432 15573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:17.452179 15573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:17.455948 15573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:17.456211 15573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:17.456960 15573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:17.458005 15573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:17.458060 15573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:17.458144 15573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:17.458184 15573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:17.467842 15573 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:38195
I20260810 18:05:17.467932 15710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:38195 every 8 connection(s)
I20260810 18:05:17.468860 15573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:05:17.477949 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 15573
I20260810 18:05:17.478201 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:05:17.478619 29005 external_mini_cluster.cc:1428] Setting key 518c2bf7cc9c7c2ff233db5bd93f032b
Aug 10 18:05:17 dist-test-slave-srgv krb5kdc[15181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385117, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:17.483680 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.470850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:56319 (local address 127.28.83.126:42171)
0810 18:05:17.470998 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.471002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.472023 (+ 1021us) server_negotiation.cc:407] Connection header received
0810 18:05:17.472965 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.472969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.473032 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.473146 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:17.474420 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.474964 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.476344 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.476571 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.476694 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.480056 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:17.480065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:17.480067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:17.480094 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:17.481743 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:17.482287 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:17.482289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:17.482290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:17.482367 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:17.482706 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:17.482709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:17.482710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:17.482895 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:17.482970 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.483454 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.483559 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:05:17.484679 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.471159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:42171 (local address 127.28.83.67:56319)
0810 18:05:17.471854 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:17.471899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:17.472734 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:17.473339 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:17.473346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:17.473749 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:17.474251 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.474267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.475110 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.475114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.476205 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.476216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.476750 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.476763 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.476948 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.477688 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:17.477707 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:17.479849 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:17.481861 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:17.481867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:17.481881 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:17.482183 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:17.482461 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:17.482463 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:17.482464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:17.482626 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:17.482984 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:17.482989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:17.483246 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 18:05:17.483407 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 18:05:17.483694 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":125,"threads_started":1}
I20260810 18:05:17.485922 15711 heartbeater.cc:347] Connected to a master server at 127.28.83.126:42171
I20260810 18:05:17.486143 15711 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:17.486608 15711 heartbeater.cc:511] Master 127.28.83.126:42171 requested a full tablet report, sending...
I20260810 18:05:17.487663 15228 ts_manager.cc:194] Registered new tserver with Master: dccce2cdf2a843a8895dd4ad30f4f85c (127.28.83.67:38195)
I20260810 18:05:17.488294 15228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:56319
I20260810 18:05:17.492836 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:05:17.502940 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.495051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.28.83.126:42171)
0810 18:05:17.495177 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.495181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.495335 (+ 154us) server_negotiation.cc:407] Connection header received
0810 18:05:17.495439 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.495442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.495482 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.495553 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:17.496281 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.496784 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.497683 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.497890 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.497969 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.499620 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:17.499623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:17.499625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:17.499647 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:17.500911 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:17.501548 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:17.501551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:17.501552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:17.501632 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:17.501985 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:17.501988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:17.501989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:17.502334 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:17.502472 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.502720 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.502812 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:05:17.506394 15228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:05:17.509538 15228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:05:17.525658 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.518717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.28.83.67:38195)
0810 18:05:17.520261 (+ 1544us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.520272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.520292 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:05:17.520376 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.520381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.520536 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.521051 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:17.521239 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.522350 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.524149 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.525009 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.525115 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.525254 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.525357 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.525496 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":146,"threads_started":1}
I20260810 18:05:17.525808 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.519472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.28.83.66:42179)
0810 18:05:17.520173 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.520188 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.520214 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:05:17.520306 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.520310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.520495 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.521062 (+ 567us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:17.521211 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.522026 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.523871 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.524802 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.524925 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.525103 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.525526 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.525644 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":135,"threads_started":1}
I20260810 18:05:17.525871 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.518554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:38195 (local address 127.0.0.1:36318)
0810 18:05:17.519263 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:17.519336 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:17.519498 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:17.520702 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:17.520705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:17.520739 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:17.520879 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.520885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.522496 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.522499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.523991 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.524003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.525132 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.525135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.525231 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.525261 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:05:17.525326 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:05:17.525434 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":108,"spinlock_wait_cycles":20352,"thread_start_us":106,"threads_started":1}
I20260810 18:05:17.526328 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.518698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:42179 (local address 127.0.0.1:46342)
0810 18:05:17.519566 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:17.519581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:17.519704 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:17.520657 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:17.520660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:17.520671 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:17.520913 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.520921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.522254 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.522258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.523727 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.523737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.524937 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.524940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.525028 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.525121 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 18:05:17.525247 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 18:05:17.525443 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":53,"spinlock_wait_cycles":4736,"thread_start_us":129,"threads_started":1}
I20260810 18:05:17.527660 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.518966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42113 (local address 127.0.0.1:49374)
0810 18:05:17.519893 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:17.519911 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:17.520000 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:17.520629 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:17.520639 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:17.520664 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:17.520913 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.520921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.522494 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.522497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.524002 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.524008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.527282 (+ 3274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.527286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.527373 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.527396 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:05:17.527469 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 18:05:17.527545 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":244,"thread_start_us":105,"threads_started":1}
I20260810 18:05:17.528242 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.519758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.28.83.65:42113)
0810 18:05:17.520103 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.520112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.520138 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:05:17.520221 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.520226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.520455 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.520627 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:17.521055 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.522341 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.525196 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.526951 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.527065 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.527201 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.527995 (+ 794us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.528102 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
I20260810 18:05:17.529069 15639 tablet_service.cc:1467] Processing CreateTablet for tablet f9537e224f6e40b8af19b93fa583cc2b (DEFAULT_TABLE table=test-table [id=000efa8e7783427aa09651acb9f11c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:17.528960 15494 tablet_service.cc:1467] Processing CreateTablet for tablet f9537e224f6e40b8af19b93fa583cc2b (DEFAULT_TABLE table=test-table [id=000efa8e7783427aa09651acb9f11c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:17.530297 15494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9537e224f6e40b8af19b93fa583cc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:17.530310 15639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9537e224f6e40b8af19b93fa583cc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:17.531560 15356 tablet_service.cc:1467] Processing CreateTablet for tablet f9537e224f6e40b8af19b93fa583cc2b (DEFAULT_TABLE table=test-table [id=000efa8e7783427aa09651acb9f11c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:17.532783 15356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9537e224f6e40b8af19b93fa583cc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:17.537750 15729 tablet_bootstrap.cc:492] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: Bootstrap starting.
I20260810 18:05:17.540258 15729 tablet_bootstrap.cc:654] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:17.540282 15731 tablet_bootstrap.cc:492] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db: Bootstrap starting.
I20260810 18:05:17.540949 15730 tablet_bootstrap.cc:492] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: Bootstrap starting.
I20260810 18:05:17.541957 15729 log.cc:824] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:17.542786 15731 tablet_bootstrap.cc:654] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:17.543188 15730 tablet_bootstrap.cc:654] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:17.543776 15731 log.cc:824] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:17.544131 15729 tablet_bootstrap.cc:492] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: No bootstrap required, opened a new log
I20260810 18:05:17.544363 15729 ts_tablet_manager.cc:1397] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:05:17.544452 15730 log.cc:824] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:17.545965 15731 tablet_bootstrap.cc:492] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db: No bootstrap required, opened a new log
I20260810 18:05:17.546224 15731 ts_tablet_manager.cc:1397] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:05:17.546554 15730 tablet_bootstrap.cc:492] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: No bootstrap required, opened a new log
I20260810 18:05:17.546754 15730 ts_tablet_manager.cc:1397] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 18:05:17.547812 15729 raft_consensus.cc:348] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } }
I20260810 18:05:17.548097 15729 raft_consensus.cc:374] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:17.548161 15729 raft_consensus.cc:729] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccce2cdf2a843a8895dd4ad30f4f85c, State: Initialized, Role: FOLLOWER
I20260810 18:05:17.548687 15729 consensus_queue.cc:260] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } }
I20260810 18:05:17.549845 15729 ts_tablet_manager.cc:1428] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 18:05:17.550227 15711 heartbeater.cc:503] Master 127.28.83.126:42171 was elected leader, sending a full tablet report...
I20260810 18:05:17.550989 15730 raft_consensus.cc:348] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } }
I20260810 18:05:17.550989 15731 raft_consensus.cc:348] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } }
I20260810 18:05:17.551288 15731 raft_consensus.cc:374] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:17.551288 15730 raft_consensus.cc:374] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:17.551352 15731 raft_consensus.cc:729] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df70cd944c347d4b4d9fe81c2d196db, State: Initialized, Role: FOLLOWER
I20260810 18:05:17.551360 15730 raft_consensus.cc:729] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d08ccefa54a4fe98170d16bf1fe0a1b, State: Initialized, Role: FOLLOWER
I20260810 18:05:17.551858 15730 consensus_queue.cc:260] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } }
I20260810 18:05:17.551858 15731 consensus_queue.cc:260] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } }
I20260810 18:05:17.552938 15730 ts_tablet_manager.cc:1428] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:05:17.552938 15731 ts_tablet_manager.cc:1428] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:05:17.553300 15568 heartbeater.cc:503] Master 127.28.83.126:42171 was elected leader, sending a full tablet report...
I20260810 18:05:17.553300 15426 heartbeater.cc:503] Master 127.28.83.126:42171 was elected leader, sending a full tablet report...
I20260810 18:05:17.649557 15737 raft_consensus.cc:482] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:17.649873 15737 raft_consensus.cc:504] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } }
I20260810 18:05:17.651064 15737 leader_election.cc:288] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dccce2cdf2a843a8895dd4ad30f4f85c (127.28.83.67:38195), 9d08ccefa54a4fe98170d16bf1fe0a1b (127.28.83.66:42179)
I20260810 18:05:17.655493 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.651569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:34113 (local address 127.28.83.67:38195)
0810 18:05:17.651729 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.651734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.652157 (+ 423us) server_negotiation.cc:407] Connection header received
0810 18:05:17.652377 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.652380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.652434 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.652524 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:17.652807 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.653309 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.654440 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.654873 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.655012 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.655070 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.655250 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.655330 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:05:17.655514 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.651581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:41361 (local address 127.28.83.66:42179)
0810 18:05:17.651732 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.651737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.652016 (+ 279us) server_negotiation.cc:407] Connection header received
0810 18:05:17.652231 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.652234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.652289 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.652385 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:17.652678 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.653186 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.654476 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.654914 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.655056 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.655112 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.655309 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.655383 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 18:05:17.655547 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.651432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:42179 (local address 127.28.83.65:41361)
0810 18:05:17.651909 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:17.651922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:17.652075 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:17.652397 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:17.652400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:17.652416 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:17.652553 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.652559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.653310 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.653314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.654345 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.654355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.655049 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.655053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.655172 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.655199 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:05:17.655281 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:05:17.655354 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":47,"thread_start_us":177,"threads_started":1}
I20260810 18:05:17.655767 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.651432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:38195 (local address 127.28.83.65:34113)
0810 18:05:17.652043 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:17.652060 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:17.652243 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:17.652536 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:17.652539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:17.652556 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:17.652684 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.652690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.653450 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.653455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.654307 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:17.654317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.655011 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.655015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:17.655111 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.655134 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:05:17.655234 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 18:05:17.655311 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":66,"thread_start_us":133,"threads_started":1}
I20260810 18:05:17.656338 15517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9537e224f6e40b8af19b93fa583cc2b" candidate_uuid: "7df70cd944c347d4b4d9fe81c2d196db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" is_pre_election: true
I20260810 18:05:17.656478 15661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9537e224f6e40b8af19b93fa583cc2b" candidate_uuid: "7df70cd944c347d4b4d9fe81c2d196db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" is_pre_election: true
I20260810 18:05:17.656718 15517 raft_consensus.cc:2454] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df70cd944c347d4b4d9fe81c2d196db in term 0.
I20260810 18:05:17.656718 15661 raft_consensus.cc:2454] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7df70cd944c347d4b4d9fe81c2d196db in term 0.
I20260810 18:05:17.657255 15310 leader_election.cc:302] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7df70cd944c347d4b4d9fe81c2d196db, 9d08ccefa54a4fe98170d16bf1fe0a1b, dccce2cdf2a843a8895dd4ad30f4f85c; no voters:
I20260810 18:05:17.657577 15737 raft_consensus.cc:2781] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:05:17.657656 15737 raft_consensus.cc:482] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:05:17.657687 15737 raft_consensus.cc:3037] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:17.658809 15737 raft_consensus.cc:504] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } }
I20260810 18:05:17.659232 15737 leader_election.cc:288] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [CANDIDATE]: Term 1 election: Requested vote from peers dccce2cdf2a843a8895dd4ad30f4f85c (127.28.83.67:38195), 9d08ccefa54a4fe98170d16bf1fe0a1b (127.28.83.66:42179)
I20260810 18:05:17.659655 15661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9537e224f6e40b8af19b93fa583cc2b" candidate_uuid: "7df70cd944c347d4b4d9fe81c2d196db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c"
I20260810 18:05:17.659667 15517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9537e224f6e40b8af19b93fa583cc2b" candidate_uuid: "7df70cd944c347d4b4d9fe81c2d196db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b"
I20260810 18:05:17.659803 15517 raft_consensus.cc:3037] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:17.659811 15661 raft_consensus.cc:3037] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:17.660954 15517 raft_consensus.cc:2454] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df70cd944c347d4b4d9fe81c2d196db in term 1.
I20260810 18:05:17.660960 15661 raft_consensus.cc:2454] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7df70cd944c347d4b4d9fe81c2d196db in term 1.
I20260810 18:05:17.661335 15310 leader_election.cc:302] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df70cd944c347d4b4d9fe81c2d196db, 9d08ccefa54a4fe98170d16bf1fe0a1b; no voters:
I20260810 18:05:17.661556 15737 raft_consensus.cc:2781] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:17.661832 15737 raft_consensus.cc:686] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 1 LEADER]: Becoming Leader. State: Replica: 7df70cd944c347d4b4d9fe81c2d196db, State: Running, Role: LEADER
I20260810 18:05:17.662225 15737 consensus_queue.cc:237] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } }
I20260810 18:05:17.665915 15227 catalog_manager.cc:5495] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db reported cstate change: term changed from 0 to 1, leader changed from <none> to 7df70cd944c347d4b4d9fe81c2d196db (127.28.83.65). New cstate: current_term: 1 leader_uuid: "7df70cd944c347d4b4d9fe81c2d196db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df70cd944c347d4b4d9fe81c2d196db" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42113 } health_report { overall_health: HEALTHY } } }
I20260810 18:05:17.681380 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.677397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.28.83.65:42113)
0810 18:05:17.677618 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.677625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.677642 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:05:17.677726 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.677728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.677783 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.677899 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:17.678206 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.678800 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.679714 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.679942 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.680058 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.680414 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:17.680858 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:17.681034 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.681156 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.681208 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 18:05:17.688225 15517 raft_consensus.cc:1261] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [term 1 FOLLOWER]: Refusing update from remote peer 7df70cd944c347d4b4d9fe81c2d196db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:17.688225 15661 raft_consensus.cc:1261] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [term 1 FOLLOWER]: Refusing update from remote peer 7df70cd944c347d4b4d9fe81c2d196db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:17.689034 15740 consensus_queue.cc:1035] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [LEADER]: Connected to new peer: Peer: permanent_uuid: "dccce2cdf2a843a8895dd4ad30f4f85c" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:17.689229 15737 consensus_queue.cc:1035] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d08ccefa54a4fe98170d16bf1fe0a1b" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:17.702704 15744 mvcc.cc:205] Tried to move back new op lower bound from 7317033442042142720 to 7317033441947250688. Current Snapshot: MvccSnapshot[applied={T|T < 7317033442042142720}]
I20260810 18:05:17.707617 15227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:17.707825 15227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:17.710450 15227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17e09747a4b3409f9a223438ef047d77: Sending DeleteTablet for 3 replicas of tablet f9537e224f6e40b8af19b93fa583cc2b
I20260810 18:05:17.711472 15639 tablet_service.cc:1514] Processing DeleteTablet for tablet f9537e224f6e40b8af19b93fa583cc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:17 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:36318
I20260810 18:05:17.711759 15356 tablet_service.cc:1514] Processing DeleteTablet for tablet f9537e224f6e40b8af19b93fa583cc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:17 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:49374
I20260810 18:05:17.711779 15494 tablet_service.cc:1514] Processing DeleteTablet for tablet f9537e224f6e40b8af19b93fa583cc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:17 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:46342
I20260810 18:05:17.712289 15754 tablet_replica.cc:330] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: stopping tablet replica
I20260810 18:05:17.712698 15754 raft_consensus.cc:2229] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:17.713141 15754 raft_consensus.cc:2258] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:17.713263 15753 tablet_replica.cc:330] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db: stopping tablet replica
I20260810 18:05:17.713377 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 15291
I20260810 18:05:17.713620 15753 raft_consensus.cc:2229] T f9537e224f6e40b8af19b93fa583cc2b P 7df70cd944c347d4b4d9fe81c2d196db [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:05:17.714155 15752 tablet_replica.cc:330] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: stopping tablet replica
I20260810 18:05:17.714466 15752 raft_consensus.cc:2229] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:17.714812 15752 raft_consensus.cc:2258] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:17.716464 15754 ts_tablet_manager.cc:1905] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:17.718111 15752 ts_tablet_manager.cc:1905] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:17.719642 15754 ts_tablet_manager.cc:1918] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:17.719769 15754 log.cc:1196] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/f9537e224f6e40b8af19b93fa583cc2b
I20260810 18:05:17.720145 15754 ts_tablet_manager.cc:1939] T f9537e224f6e40b8af19b93fa583cc2b P 9d08ccefa54a4fe98170d16bf1fe0a1b: Deleting consensus metadata
I20260810 18:05:17.721061 15752 ts_tablet_manager.cc:1918] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:17.721174 15752 log.cc:1196] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/f9537e224f6e40b8af19b93fa583cc2b
I20260810 18:05:17.721617 15752 ts_tablet_manager.cc:1939] T f9537e224f6e40b8af19b93fa583cc2b P dccce2cdf2a843a8895dd4ad30f4f85c: Deleting consensus metadata
I20260810 18:05:17.721709 15215 catalog_manager.cc:4841] TS 9d08ccefa54a4fe98170d16bf1fe0a1b (127.28.83.66:42179): tablet f9537e224f6e40b8af19b93fa583cc2b (table test-table [id=000efa8e7783427aa09651acb9f11c31]) successfully deleted
W20260810 18:05:17.722450 15213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:42113 (error 108)
I20260810 18:05:17.723058 15214 catalog_manager.cc:4841] TS dccce2cdf2a843a8895dd4ad30f4f85c (127.28.83.67:38195): tablet f9537e224f6e40b8af19b93fa583cc2b (table test-table [id=000efa8e7783427aa09651acb9f11c31]) successfully deleted
I20260810 18:05:17.723577 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.723296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42113 (local address 127.0.0.1:49384)
0810 18:05:17.723427 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:17.723494 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 18:05:17.723731 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 15431
W20260810 18:05:17.723769 15213 catalog_manager.cc:4584] TS 7df70cd944c347d4b4d9fe81c2d196db (127.28.83.65:42113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9537e224f6e40b8af19b93fa583cc2b: Network error: Client connection negotiation failed: client connection to 127.28.83.65:42113: connect: Connection refused (error 111)
I20260810 18:05:17.730701 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 15573
I20260810 18:05:17.737164 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 15195
2026-08-10T18:05:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2431 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T18:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:17Z Disabled control of system clock
I20260810 18:05:17.770051 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40567
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:40567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:05:17.883653 15759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:17.883961 15759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:17.883999 15759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:17.888042 15759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:17.888106 15759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:17.888123 15759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:17.888139 15759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:17.888154 15759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:17.892978 15759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:40567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:17.894109 15759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:17.895485 15759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:17.904450 15765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:17.904539 15767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:17.904685 15759 server_base.cc:1034] running on GCE node
W20260810 18:05:17.904445 15764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:17.905444 15759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:17.906672 15759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:17.907861 15759 hybrid_clock.cc:648] HybridClock initialized: now 1786385117907837 us; error 58 us; skew 500 ppm
I20260810 18:05:17.909950 15759 webserver.cc:460] Webserver started at http://127.28.83.126:33687/ using document root <none> and password file <none>
I20260810 18:05:17.910542 15759 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:17.910646 15759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:17.910887 15759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:17.912987 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "4a03b0b0af994de998b9af132e40aa9c"
format_stamp: "Formatted at 2026-08-10 18:05:17 on dist-test-slave-srgv"
server_key: "8beff645c57ecf2f03eb8ffceef28da5"
server_key_iv: "21a7ec21f79c86d559379ca32c5ea625"
server_key_version: "encryptionkey@0"
I20260810 18:05:17.913647 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "4a03b0b0af994de998b9af132e40aa9c"
format_stamp: "Formatted at 2026-08-10 18:05:17 on dist-test-slave-srgv"
server_key: "8beff645c57ecf2f03eb8ffceef28da5"
server_key_iv: "21a7ec21f79c86d559379ca32c5ea625"
server_key_version: "encryptionkey@0"
I20260810 18:05:17.917330 15759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:05:17.919857 15773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:17.921092 15759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:05:17.921247 15759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "4a03b0b0af994de998b9af132e40aa9c"
format_stamp: "Formatted at 2026-08-10 18:05:17 on dist-test-slave-srgv"
server_key: "8beff645c57ecf2f03eb8ffceef28da5"
server_key_iv: "21a7ec21f79c86d559379ca32c5ea625"
server_key_version: "encryptionkey@0"
I20260810 18:05:17.921371 15759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:17.950700 15759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:17.951607 15759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:17.951787 15759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:17.959537 15759 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:40567
I20260810 18:05:17.959561 15827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:40567 every 8 connection(s)
I20260810 18:05:17.960552 15759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:17.963743 15829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:17.966439 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 15759
I20260810 18:05:17.966549 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:17.966862 29005 external_mini_cluster.cc:1428] Setting key a1c5dc6fef54e50529c1a5d6c4d8a78f
I20260810 18:05:17.969933 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c: Bootstrap starting.
I20260810 18:05:17.972546 15829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:17.973489 15829 log.cc:824] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:17.975906 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c: No bootstrap required, opened a new log
I20260810 18:05:17.976619 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:17.968428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.28.83.126:40567)
0810 18:05:17.969070 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:17.969086 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:17.969123 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:05:17.969826 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:17.969845 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:17.970296 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:17.970579 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:17.970956 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.971917 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.973104 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:17.973720 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:17.973843 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:17.974809 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:17.974814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:17.974827 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:17.975218 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:17.975271 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:17.975290 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:17.975368 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:17.975537 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 18:05:17.975848 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":140,"threads_started":1}
I20260810 18:05:17.979979 15829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a03b0b0af994de998b9af132e40aa9c" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40567 } }
I20260810 18:05:17.980225 15829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:17.980260 15829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a03b0b0af994de998b9af132e40aa9c, State: Initialized, Role: FOLLOWER
I20260810 18:05:17.980683 15829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a03b0b0af994de998b9af132e40aa9c" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40567 } }
I20260810 18:05:17.980806 15829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:17.980852 15829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:17.980922 15829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:17.981974 15829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a03b0b0af994de998b9af132e40aa9c" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40567 } }
I20260810 18:05:17.982295 15829 leader_election.cc:302] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a03b0b0af994de998b9af132e40aa9c; no voters:
I20260810 18:05:17.982679 15829 leader_election.cc:288] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:17.982762 15834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:17.982954 15834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [term 1 LEADER]: Becoming Leader. State: Replica: 4a03b0b0af994de998b9af132e40aa9c, State: Running, Role: LEADER
I20260810 18:05:17.983250 15834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a03b0b0af994de998b9af132e40aa9c" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40567 } }
I20260810 18:05:17.984012 15829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:17.985107 15836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a03b0b0af994de998b9af132e40aa9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a03b0b0af994de998b9af132e40aa9c" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40567 } } }
I20260810 18:05:17.985244 15836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:17.985522 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a03b0b0af994de998b9af132e40aa9c. Latest consensus state: current_term: 1 leader_uuid: "4a03b0b0af994de998b9af132e40aa9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a03b0b0af994de998b9af132e40aa9c" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40567 } } }
I20260810 18:05:17.985615 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:17.986407 15844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:17.989108 15844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:17.994581 15844 catalog_manager.cc:1347] Generated new cluster ID: fdef663f8ee9408b83db637c8cee2459
I20260810 18:05:17.994664 15844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:18.020906 15844 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:18.022135 15844 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:18.040345 15844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c: Generated new TSK 0
I20260810 18:05:18.041163 15844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:18.044225 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40567
--builtin_ntp_servers=127.28.83.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:05:18.162931 15855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:18.163235 15855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:18.163277 15855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:18.167518 15855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:18.167599 15855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:18.167660 15855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:05:18.172888 15855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:18.174152 15855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:18.176204 15855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:18.186883 15863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:18.186892 15860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:18.186902 15861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:18.187103 15855 server_base.cc:1034] running on GCE node
I20260810 18:05:18.187747 15855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:18.188436 15855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:18.189641 15855 hybrid_clock.cc:648] HybridClock initialized: now 1786385118189629 us; error 27 us; skew 500 ppm
I20260810 18:05:18.191735 15855 webserver.cc:460] Webserver started at http://127.28.83.65:35555/ using document root <none> and password file <none>
I20260810 18:05:18.192378 15855 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:18.192478 15855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:18.192744 15855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:18.195037 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "68b0c30ce17f4835bfbbce73183c8e99"
format_stamp: "Formatted at 2026-08-10 18:05:18 on dist-test-slave-srgv"
server_key: "7f921444af24aa36dd25bb26a2d90392"
server_key_iv: "b16c5303d0216789e7bfdb16613b033d"
server_key_version: "encryptionkey@0"
I20260810 18:05:18.195674 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "68b0c30ce17f4835bfbbce73183c8e99"
format_stamp: "Formatted at 2026-08-10 18:05:18 on dist-test-slave-srgv"
server_key: "7f921444af24aa36dd25bb26a2d90392"
server_key_iv: "b16c5303d0216789e7bfdb16613b033d"
server_key_version: "encryptionkey@0"
I20260810 18:05:18.199708 15855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:18.202414 15869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:18.203909 15855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 18:05:18.204068 15855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "68b0c30ce17f4835bfbbce73183c8e99"
format_stamp: "Formatted at 2026-08-10 18:05:18 on dist-test-slave-srgv"
server_key: "7f921444af24aa36dd25bb26a2d90392"
server_key_iv: "b16c5303d0216789e7bfdb16613b033d"
server_key_version: "encryptionkey@0"
I20260810 18:05:18.204195 15855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:18.226115 15855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:18.226892 15855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:18.227105 15855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:18.227758 15855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:18.228753 15855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:18.228801 15855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:18.228874 15855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:18.228906 15855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:18.238680 15855 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:41709
I20260810 18:05:18.238700 15987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:41709 every 8 connection(s)
I20260810 18:05:18.239642 15855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:05:18.241721 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 15855
I20260810 18:05:18.241859 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:05:18.242182 29005 external_mini_cluster.cc:1428] Setting key 55b83e6e850e801cf70f910c88f329b8
I20260810 18:05:18.244225 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40567
--builtin_ntp_servers=127.28.83.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:05:18.250552 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.241514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:59991 (local address 127.28.83.126:40567)
0810 18:05:18.241662 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.241666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.242930 (+ 1264us) server_negotiation.cc:407] Connection header received
0810 18:05:18.244557 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.244561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.244619 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.244701 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:18.245771 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.246342 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.248198 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.248418 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.248496 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.250056 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:18.250058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:18.250061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:18.250074 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:18.250092 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:18.250100 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:18.250161 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.250377 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.250433 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:05:18.251914 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.241975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40567 (local address 127.28.83.65:59991)
0810 18:05:18.242775 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.242833 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.244297 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.244876 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.244882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.244966 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:18.245579 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.245605 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.246488 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.246495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.248064 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.248079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.248607 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.248614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.248874 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.249696 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:05:18.249701 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:18.249767 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:05:18.249874 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:18.250194 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:18.250207 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:18.250215 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.250359 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.250719 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":138,"threads_started":1}
I20260810 18:05:18.253281 15988 heartbeater.cc:347] Connected to a master server at 127.28.83.126:40567
I20260810 18:05:18.253566 15988 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:18.254122 15988 heartbeater.cc:511] Master 127.28.83.126:40567 requested a full tablet report, sending...
I20260810 18:05:18.255661 15785 ts_manager.cc:194] Registered new tserver with Master: 68b0c30ce17f4835bfbbce73183c8e99 (127.28.83.65:41709)
I20260810 18:05:18.256812 15785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.65:59991
W20260810 18:05:18.362188 15991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:18.362509 15991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:18.362581 15991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:18.366771 15991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:18.366870 15991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:18.366966 15991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:05:18.372048 15991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:18.373252 15991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:18.374760 15991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:18.384852 15996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:18.384922 15997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:18.384963 15991 server_base.cc:1034] running on GCE node
W20260810 18:05:18.384835 15999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:18.385638 15991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:18.386369 15991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:18.387569 15991 hybrid_clock.cc:648] HybridClock initialized: now 1786385118387535 us; error 51 us; skew 500 ppm
I20260810 18:05:18.389605 15991 webserver.cc:460] Webserver started at http://127.28.83.66:38719/ using document root <none> and password file <none>
I20260810 18:05:18.390193 15991 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:18.390297 15991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:18.390532 15991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:18.392635 15991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "2e1e5da611434d469789f9e9985c9c9a"
format_stamp: "Formatted at 2026-08-10 18:05:18 on dist-test-slave-srgv"
server_key: "d22fcbbcdbb1072cb16fc839a507db43"
server_key_iv: "c54856598669007ce80d312232d59fbb"
server_key_version: "encryptionkey@0"
I20260810 18:05:18.393224 15991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "2e1e5da611434d469789f9e9985c9c9a"
format_stamp: "Formatted at 2026-08-10 18:05:18 on dist-test-slave-srgv"
server_key: "d22fcbbcdbb1072cb16fc839a507db43"
server_key_iv: "c54856598669007ce80d312232d59fbb"
server_key_version: "encryptionkey@0"
I20260810 18:05:18.397032 15991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:05:18.399668 16005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:18.400960 15991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:05:18.401108 15991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "2e1e5da611434d469789f9e9985c9c9a"
format_stamp: "Formatted at 2026-08-10 18:05:18 on dist-test-slave-srgv"
server_key: "d22fcbbcdbb1072cb16fc839a507db43"
server_key_iv: "c54856598669007ce80d312232d59fbb"
server_key_version: "encryptionkey@0"
I20260810 18:05:18.401237 15991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:18.422421 15991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:18.423215 15991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:18.423444 15991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:18.424094 15991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:18.425201 15991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:18.425275 15991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:18.425343 15991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:18.425392 15991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:18.434470 15991 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:42195
I20260810 18:05:18.434582 16124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:42195 every 8 connection(s)
I20260810 18:05:18.435503 15991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:05:18.442456 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 15991
I20260810 18:05:18.442602 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:05:18.442925 29005 external_mini_cluster.cc:1428] Setting key f805e196f19b2d069b45e2138f2df169
I20260810 18:05:18.445186 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40567
--builtin_ntp_servers=127.28.83.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:05:18.445852 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.437523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:49227 (local address 127.28.83.126:40567)
0810 18:05:18.437681 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.437685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.438855 (+ 1170us) server_negotiation.cc:407] Connection header received
0810 18:05:18.440268 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.440271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.440321 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.440401 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:18.441360 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.441950 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.443604 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.443860 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.443977 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.445232 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:18.445235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:18.445237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:18.445249 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:18.445298 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:18.445309 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:18.445390 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.445630 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.445698 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:05:18.446784 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.437984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40567 (local address 127.28.83.66:49227)
0810 18:05:18.438685 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.438728 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.440052 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.440585 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.440592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.440682 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:18.441224 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.441238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.442071 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.442074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.443456 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.443467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.444025 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.444029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.444223 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.444880 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:05:18.444883 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:18.444947 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:05:18.445040 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:18.445466 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:18.445483 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:18.445488 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.445628 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.445946 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":127,"threads_started":1}
I20260810 18:05:18.448086 16125 heartbeater.cc:347] Connected to a master server at 127.28.83.126:40567
I20260810 18:05:18.448354 16125 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:18.448927 16125 heartbeater.cc:511] Master 127.28.83.126:40567 requested a full tablet report, sending...
I20260810 18:05:18.450117 15785 ts_manager.cc:194] Registered new tserver with Master: 2e1e5da611434d469789f9e9985c9c9a (127.28.83.66:42195)
I20260810 18:05:18.451032 15785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.66:49227
W20260810 18:05:18.564438 16128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:18.564743 16128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:18.564786 16128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:18.569098 16128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:18.569185 16128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:18.569244 16128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:05:18.574962 16128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:40567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:18.576088 16128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:18.577610 16128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:18.587903 16133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:18.587903 16134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:18.587924 16136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:18.588249 16128 server_base.cc:1034] running on GCE node
I20260810 18:05:18.588728 16128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:18.589509 16128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:18.590703 16128 hybrid_clock.cc:648] HybridClock initialized: now 1786385118590660 us; error 70 us; skew 500 ppm
I20260810 18:05:18.592741 16128 webserver.cc:460] Webserver started at http://127.28.83.67:38599/ using document root <none> and password file <none>
I20260810 18:05:18.593528 16128 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:18.593628 16128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:18.593869 16128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:18.596143 16128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "c032af05a4224d24989c591c06a8294d"
format_stamp: "Formatted at 2026-08-10 18:05:18 on dist-test-slave-srgv"
server_key: "a5c32424601989ebd93453a512a8e7e4"
server_key_iv: "02773a2ceae990f1966e7c57bce5d361"
server_key_version: "encryptionkey@0"
I20260810 18:05:18.596827 16128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "c032af05a4224d24989c591c06a8294d"
format_stamp: "Formatted at 2026-08-10 18:05:18 on dist-test-slave-srgv"
server_key: "a5c32424601989ebd93453a512a8e7e4"
server_key_iv: "02773a2ceae990f1966e7c57bce5d361"
server_key_version: "encryptionkey@0"
I20260810 18:05:18.601233 16128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:05:18.604280 16142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:18.605854 16128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:18.606029 16128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "c032af05a4224d24989c591c06a8294d"
format_stamp: "Formatted at 2026-08-10 18:05:18 on dist-test-slave-srgv"
server_key: "a5c32424601989ebd93453a512a8e7e4"
server_key_iv: "02773a2ceae990f1966e7c57bce5d361"
server_key_version: "encryptionkey@0"
I20260810 18:05:18.606178 16128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:18.648146 16128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:18.648955 16128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:18.649195 16128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:18.649922 16128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:18.650942 16128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:18.651015 16128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:18.651090 16128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:18.651144 16128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:18.660189 16128 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:38657
I20260810 18:05:18.660243 16260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:38657 every 8 connection(s)
I20260810 18:05:18.661199 16128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:05:18.663578 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 16128
I20260810 18:05:18.663700 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:05:18.664036 29005 external_mini_cluster.cc:1428] Setting key 8fe90e0e4a33a3c1f31e798f3882cdce
I20260810 18:05:18.671495 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.663215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:45111 (local address 127.28.83.126:40567)
0810 18:05:18.663365 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.663369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.664791 (+ 1422us) server_negotiation.cc:407] Connection header received
0810 18:05:18.666536 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.666539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.666601 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.666713 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:18.667653 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.668192 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.669441 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.669691 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.669812 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.670976 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:18.670982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:18.670989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:18.671002 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:18.671021 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:18.671029 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:18.671109 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.671281 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.671347 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:05:18.672390 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.663735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:40567 (local address 127.28.83.67:45111)
0810 18:05:18.664614 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.664679 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.666268 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.666903 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.666911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.667000 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:18.667510 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.667522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.668332 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.668335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.669267 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.669277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.669869 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.669874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.670058 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.670698 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:05:18.670700 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:18.670742 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:05:18.670813 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:18.671110 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:18.671123 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:18.671130 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.671284 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.671587 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":149,"threads_started":1}
I20260810 18:05:18.673686 16261 heartbeater.cc:347] Connected to a master server at 127.28.83.126:40567
I20260810 18:05:18.673915 16261 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:18.674424 16261 heartbeater.cc:511] Master 127.28.83.126:40567 requested a full tablet report, sending...
I20260810 18:05:18.675449 15785 ts_manager.cc:194] Registered new tserver with Master: c032af05a4224d24989c591c06a8294d (127.28.83.67:38657)
I20260810 18:05:18.675987 15785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.67:45111
I20260810 18:05:18.678282 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:05:18.684461 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.680316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.28.83.126:40567)
0810 18:05:18.680445 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.680448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.680628 (+ 180us) server_negotiation.cc:407] Connection header received
0810 18:05:18.680734 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.680736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.680784 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.680853 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:18.681324 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.681906 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.682787 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.682995 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.683070 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.683993 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:18.683996 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:18.683998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:18.684011 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:18.684029 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:18.684036 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:18.684150 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.684304 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.684342 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:05:18.687868 15785 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:05:18.691026 15785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:05:18.706993 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.701001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.28.83.65:41709)
0810 18:05:18.701821 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.701835 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.701857 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:05:18.701929 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.701936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.702121 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.702281 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:18.702706 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.703760 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.705490 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.706239 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.706371 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.706594 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.706710 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.706816 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":135,"threads_started":1}
I20260810 18:05:18.706988 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.700799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:41709 (local address 127.0.0.1:44218)
0810 18:05:18.701173 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.701189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.701289 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.702450 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.702456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.702476 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:18.702571 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.702577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.703877 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.703882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.705270 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.705282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.706371 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.706375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.706504 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.706544 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.706678 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.706815 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":158,"thread_start_us":97,"threads_started":1}
I20260810 18:05:18.707333 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.700861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.28.83.67:38657)
0810 18:05:18.701696 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.701708 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.701731 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:05:18.701968 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.701973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.702121 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.702275 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:18.702492 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.703299 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.705356 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.706279 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.706397 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.706593 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.707038 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.707155 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":180,"threads_started":1}
I20260810 18:05:18.707795 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.700107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.28.83.66:42195)
0810 18:05:18.701120 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.701140 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.701177 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:05:18.701260 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.701266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.701540 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.701710 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:18.702251 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.703469 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.705564 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.706319 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.706412 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.706594 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.707579 (+ 985us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.707666 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":168,"threads_started":1}
I20260810 18:05:18.708251 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.700064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:38657 (local address 127.0.0.1:42050)
0810 18:05:18.700696 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.700712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.700811 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.702263 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.702266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.702285 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:18.702373 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.702378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.703420 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.703427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.705242 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.705253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.706662 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.706666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.706809 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.706869 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.706941 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.708127 (+ 1186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":1253,"thread_start_us":130,"threads_started":1}
I20260810 18:05:18.708516 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.699992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:42195 (local address 127.0.0.1:40336)
0810 18:05:18.700353 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.700403 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.700528 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.701903 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.701910 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.701934 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:18.702104 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.702114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.703586 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.703589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.705437 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.705444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.706662 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.706666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.706814 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.706868 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.706941 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.707039 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":422,"thread_start_us":122,"threads_started":1}
I20260810 18:05:18.710290 15917 tablet_service.cc:1467] Processing CreateTablet for tablet 4419155c2345420ea2f5ece696b5194d (DEFAULT_TABLE table=test-table [id=917dc8d3b6be4764a9becb2c176fa9d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:18.711530 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4419155c2345420ea2f5ece696b5194d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:18.711150 16191 tablet_service.cc:1467] Processing CreateTablet for tablet 4419155c2345420ea2f5ece696b5194d (DEFAULT_TABLE table=test-table [id=917dc8d3b6be4764a9becb2c176fa9d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:18.712180 16191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4419155c2345420ea2f5ece696b5194d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:18.719671 16280 tablet_bootstrap.cc:492] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99: Bootstrap starting.
I20260810 18:05:18.719910 16054 tablet_service.cc:1467] Processing CreateTablet for tablet 4419155c2345420ea2f5ece696b5194d (DEFAULT_TABLE table=test-table [id=917dc8d3b6be4764a9becb2c176fa9d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:18.721087 16054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4419155c2345420ea2f5ece696b5194d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:18.721863 16281 tablet_bootstrap.cc:492] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: Bootstrap starting.
I20260810 18:05:18.722259 16280 tablet_bootstrap.cc:654] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:18.723181 16280 log.cc:824] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:18.724275 16281 tablet_bootstrap.cc:654] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:18.725183 16281 log.cc:824] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:18.725533 16280 tablet_bootstrap.cc:492] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99: No bootstrap required, opened a new log
I20260810 18:05:18.725749 16280 ts_tablet_manager.cc:1397] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:05:18.728551 16284 tablet_bootstrap.cc:492] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: Bootstrap starting.
I20260810 18:05:18.729143 16280 raft_consensus.cc:348] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.729461 16280 raft_consensus.cc:374] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:18.729526 16280 raft_consensus.cc:729] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b0c30ce17f4835bfbbce73183c8e99, State: Initialized, Role: FOLLOWER
I20260810 18:05:18.729935 16280 consensus_queue.cc:260] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.730654 15988 heartbeater.cc:503] Master 127.28.83.126:40567 was elected leader, sending a full tablet report...
I20260810 18:05:18.730695 16281 tablet_bootstrap.cc:492] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: No bootstrap required, opened a new log
I20260810 18:05:18.730916 16281 ts_tablet_manager.cc:1397] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260810 18:05:18.730911 16280 ts_tablet_manager.cc:1428] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:05:18.734931 16281 raft_consensus.cc:348] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.735217 16281 raft_consensus.cc:374] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:18.735285 16281 raft_consensus.cc:729] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c032af05a4224d24989c591c06a8294d, State: Initialized, Role: FOLLOWER
I20260810 18:05:18.735693 16281 consensus_queue.cc:260] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.736330 16261 heartbeater.cc:503] Master 127.28.83.126:40567 was elected leader, sending a full tablet report...
I20260810 18:05:18.736577 16281 ts_tablet_manager.cc:1428] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:05:18.738436 16284 tablet_bootstrap.cc:654] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:18.739305 16284 log.cc:824] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: Log is configured to *not* fsync() on all Append() calls
W20260810 18:05:18.740835 15989 tablet.cc:2369] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:05:18.741779 16284 tablet_bootstrap.cc:492] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: No bootstrap required, opened a new log
I20260810 18:05:18.742005 16284 ts_tablet_manager.cc:1397] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20260810 18:05:18.745339 16284 raft_consensus.cc:348] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.745710 16284 raft_consensus.cc:374] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:18.745800 16284 raft_consensus.cc:729] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1e5da611434d469789f9e9985c9c9a, State: Initialized, Role: FOLLOWER
I20260810 18:05:18.746273 16284 consensus_queue.cc:260] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.747040 16125 heartbeater.cc:503] Master 127.28.83.126:40567 was elected leader, sending a full tablet report...
I20260810 18:05:18.747280 16284 ts_tablet_manager.cc:1428] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260810 18:05:18.849027 16285 raft_consensus.cc:482] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:18.849332 16285 raft_consensus.cc:504] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.850615 16285 leader_election.cc:288] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e1e5da611434d469789f9e9985c9c9a (127.28.83.66:42195), c032af05a4224d24989c591c06a8294d (127.28.83.67:38657)
I20260810 18:05:18.854727 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.851058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:40359 (local address 127.28.83.66:42195)
0810 18:05:18.851217 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.851222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.851370 (+ 148us) server_negotiation.cc:407] Connection header received
0810 18:05:18.851517 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.851520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.851576 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.851668 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:18.852001 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.852515 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.853721 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.854124 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.854256 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.854312 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.854530 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.854607 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:05:18.854753 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.850924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:42195 (local address 127.28.83.65:40359)
0810 18:05:18.851237 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.851261 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.851384 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.851681 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.851684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.851732 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:18.851877 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.851882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.852637 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.852642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.853594 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.853604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.854252 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.854264 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.854396 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.854424 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.854511 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.854583 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":1,"spinlock_wait_cycles":5120,"thread_start_us":108,"threads_started":1}
I20260810 18:05:18.855214 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.851041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:39389 (local address 127.28.83.67:38657)
0810 18:05:18.851202 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.851208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.851674 (+ 466us) server_negotiation.cc:407] Connection header received
0810 18:05:18.851859 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.851862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.851912 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.851999 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:18.852299 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.852839 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.853968 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.854424 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.854543 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.854590 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.854992 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.855097 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 18:05:18.855384 16076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4419155c2345420ea2f5ece696b5194d" candidate_uuid: "68b0c30ce17f4835bfbbce73183c8e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1e5da611434d469789f9e9985c9c9a" is_pre_election: true
I20260810 18:05:18.855597 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.850924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:38657 (local address 127.28.83.65:39389)
0810 18:05:18.851566 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.851578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.851744 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.852013 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.852016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.852032 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:18.852168 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.852173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.852963 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.852966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.853844 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.853854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.854656 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.854663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.854799 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.854821 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.855360 (+ 539us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.855440 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":259,"spinlock_wait_cycles":31744,"thread_start_us":154,"threads_started":1}
I20260810 18:05:18.855746 16076 raft_consensus.cc:2454] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68b0c30ce17f4835bfbbce73183c8e99 in term 0.
I20260810 18:05:18.856272 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4419155c2345420ea2f5ece696b5194d" candidate_uuid: "68b0c30ce17f4835bfbbce73183c8e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c032af05a4224d24989c591c06a8294d" is_pre_election: true
I20260810 18:05:18.856333 15873 leader_election.cc:302] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1e5da611434d469789f9e9985c9c9a, 68b0c30ce17f4835bfbbce73183c8e99; no voters:
I20260810 18:05:18.856549 16212 raft_consensus.cc:2454] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68b0c30ce17f4835bfbbce73183c8e99 in term 0.
I20260810 18:05:18.856647 16285 raft_consensus.cc:2781] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:05:18.856746 16285 raft_consensus.cc:482] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:05:18.856712 16286 raft_consensus.cc:482] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:18.856802 16285 raft_consensus.cc:3037] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:18.856819 16286 raft_consensus.cc:504] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.858057 16286 leader_election.cc:288] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e1e5da611434d469789f9e9985c9c9a (127.28.83.66:42195), 68b0c30ce17f4835bfbbce73183c8e99 (127.28.83.65:41709)
I20260810 18:05:18.858224 16285 raft_consensus.cc:504] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.858573 16285 leader_election.cc:288] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [CANDIDATE]: Term 1 election: Requested vote from peers 2e1e5da611434d469789f9e9985c9c9a (127.28.83.66:42195), c032af05a4224d24989c591c06a8294d (127.28.83.67:38657)
I20260810 18:05:18.858894 16076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4419155c2345420ea2f5ece696b5194d" candidate_uuid: "68b0c30ce17f4835bfbbce73183c8e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1e5da611434d469789f9e9985c9c9a"
I20260810 18:05:18.859025 16076 raft_consensus.cc:3037] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:18.859285 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4419155c2345420ea2f5ece696b5194d" candidate_uuid: "68b0c30ce17f4835bfbbce73183c8e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c032af05a4224d24989c591c06a8294d"
I20260810 18:05:18.859398 16212 raft_consensus.cc:3037] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:18.860390 16076 raft_consensus.cc:2454] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68b0c30ce17f4835bfbbce73183c8e99 in term 1.
I20260810 18:05:18.860656 16212 raft_consensus.cc:2454] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68b0c30ce17f4835bfbbce73183c8e99 in term 1.
I20260810 18:05:18.860860 15873 leader_election.cc:302] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1e5da611434d469789f9e9985c9c9a, 68b0c30ce17f4835bfbbce73183c8e99; no voters:
I20260810 18:05:18.861083 16285 raft_consensus.cc:2781] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:18.861322 16285 raft_consensus.cc:686] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 1 LEADER]: Becoming Leader. State: Replica: 68b0c30ce17f4835bfbbce73183c8e99, State: Running, Role: LEADER
I20260810 18:05:18.861681 16285 consensus_queue.cc:237] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } }
I20260810 18:05:18.862929 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.858501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:35041 (local address 127.28.83.65:41709)
0810 18:05:18.858732 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.858737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.859121 (+ 384us) server_negotiation.cc:407] Connection header received
0810 18:05:18.859162 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.859165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.859215 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.859286 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:18.859678 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.860358 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.861814 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.862374 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.862509 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.862560 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.862748 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.862817 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260810 18:05:18.862950 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.858260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:41709 (local address 127.28.83.67:35041)
0810 18:05:18.858748 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.858764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.858848 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.859390 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.859393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.859412 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:18.859528 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.859535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.860521 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.860527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.861685 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.861694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.862512 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.862516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.862628 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.862652 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.862723 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.862806 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":144,"threads_started":1}
I20260810 18:05:18.863297 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.858400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:60779 (local address 127.28.83.66:42195)
0810 18:05:18.858845 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.858849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.858862 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:05:18.858907 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.858909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.858954 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.859027 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:18.859368 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.860097 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.862078 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.862617 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.862717 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.862763 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.863096 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.863170 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366}
I20260810 18:05:18.863639 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.858223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:42195 (local address 127.28.83.67:60779)
0810 18:05:18.858618 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.858650 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.858761 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:18.859071 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:18.859074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:18.859092 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:18.859229 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.859238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.860249 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.860255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.861915 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:18.861925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.862860 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.862863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:18.862963 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.862983 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:05:18.863440 (+ 457us) client_negotiation.cc:240] Negotiation successful
0810 18:05:18.863516 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312}
I20260810 18:05:18.864151 16076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4419155c2345420ea2f5ece696b5194d" candidate_uuid: "c032af05a4224d24989c591c06a8294d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1e5da611434d469789f9e9985c9c9a" is_pre_election: true
I20260810 18:05:18.864368 16076 raft_consensus.cc:2379] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c032af05a4224d24989c591c06a8294d in current term 1: Already voted for candidate 68b0c30ce17f4835bfbbce73183c8e99 in this term.
I20260810 18:05:18.864835 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4419155c2345420ea2f5ece696b5194d" candidate_uuid: "c032af05a4224d24989c591c06a8294d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b0c30ce17f4835bfbbce73183c8e99" is_pre_election: true
I20260810 18:05:18.865477 16145 leader_election.cc:302] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c032af05a4224d24989c591c06a8294d; no voters: 2e1e5da611434d469789f9e9985c9c9a, 68b0c30ce17f4835bfbbce73183c8e99
I20260810 18:05:18.865769 16286 raft_consensus.cc:2726] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:05:18.865893 15785 catalog_manager.cc:5495] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68b0c30ce17f4835bfbbce73183c8e99 (127.28.83.65). New cstate: current_term: 1 leader_uuid: "68b0c30ce17f4835bfbbce73183c8e99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68b0c30ce17f4835bfbbce73183c8e99" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 41709 } health_report { overall_health: HEALTHY } } }
I20260810 18:05:18.901507 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:18.897661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.28.83.65:41709)
0810 18:05:18.897843 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:18.897846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:18.897859 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:05:18.897930 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:18.897932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:18.897986 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:18.898109 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:18.898391 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.898920 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.899822 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:18.900047 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:18.900164 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:18.900524 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:18.900974 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:18.901146 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:18.901247 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 18:05:18.901284 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:05:18.908358 16076 raft_consensus.cc:1261] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 1 FOLLOWER]: Refusing update from remote peer 68b0c30ce17f4835bfbbce73183c8e99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:18.908398 16212 raft_consensus.cc:1261] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 1 FOLLOWER]: Refusing update from remote peer 68b0c30ce17f4835bfbbce73183c8e99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:18.909098 16285 consensus_queue.cc:1035] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1e5da611434d469789f9e9985c9c9a" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 42195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:18.909346 16293 consensus_queue.cc:1035] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c032af05a4224d24989c591c06a8294d" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:05:18.912238 16262 tablet.cc:2369] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:05:18.917816 16296 mvcc.cc:205] Tried to move back new op lower bound from 7317033447040180224 to 7317033446860558336. Current Snapshot: MvccSnapshot[applied={T|T < 7317033447040180224}]
I20260810 18:05:18.921183 16297 mvcc.cc:205] Tried to move back new op lower bound from 7317033447040180224 to 7317033446860558336. Current Snapshot: MvccSnapshot[applied={T|T < 7317033447040180224}]
I20260810 18:05:18.927330 15791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:18.927541 15791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:18.930478 15791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a03b0b0af994de998b9af132e40aa9c: Sending DeleteTablet for 3 replicas of tablet 4419155c2345420ea2f5ece696b5194d
I20260810 18:05:18.931514 16191 tablet_service.cc:1514] Processing DeleteTablet for tablet 4419155c2345420ea2f5ece696b5194d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:18 UTC) from {username='slave'} at 127.0.0.1:42050
I20260810 18:05:18.931615 16054 tablet_service.cc:1514] Processing DeleteTablet for tablet 4419155c2345420ea2f5ece696b5194d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:18 UTC) from {username='slave'} at 127.0.0.1:40336
I20260810 18:05:18.931602 15917 tablet_service.cc:1514] Processing DeleteTablet for tablet 4419155c2345420ea2f5ece696b5194d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:18 UTC) from {username='slave'} at 127.0.0.1:44218
I20260810 18:05:18.932174 16307 tablet_replica.cc:330] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99: stopping tablet replica
I20260810 18:05:18.932543 16307 raft_consensus.cc:2229] T 4419155c2345420ea2f5ece696b5194d P 68b0c30ce17f4835bfbbce73183c8e99 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:05:18.932716 16306 tablet_replica.cc:330] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: stopping tablet replica
I20260810 18:05:18.932960 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 15855
I20260810 18:05:18.934968 16306 raft_consensus.cc:2229] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:18.935083 16308 tablet_replica.cc:330] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: stopping tablet replica
I20260810 18:05:18.935387 16308 raft_consensus.cc:2229] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:18.935544 16306 raft_consensus.cc:2258] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:18.935689 16308 raft_consensus.cc:2258] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:18.937281 16308 ts_tablet_manager.cc:1905] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:18.937348 16306 ts_tablet_manager.cc:1905] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:18.939754 16308 ts_tablet_manager.cc:1918] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:18.939894 16308 log.cc:1196] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/4419155c2345420ea2f5ece696b5194d
I20260810 18:05:18.940204 16308 ts_tablet_manager.cc:1939] T 4419155c2345420ea2f5ece696b5194d P 2e1e5da611434d469789f9e9985c9c9a: Deleting consensus metadata
I20260810 18:05:18.941108 16306 ts_tablet_manager.cc:1918] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:18.941227 16306 log.cc:1196] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/4419155c2345420ea2f5ece696b5194d
I20260810 18:05:18.941531 15777 catalog_manager.cc:4841] TS 2e1e5da611434d469789f9e9985c9c9a (127.28.83.66:42195): tablet 4419155c2345420ea2f5ece696b5194d (table test-table [id=917dc8d3b6be4764a9becb2c176fa9d6]) successfully deleted
I20260810 18:05:18.941612 16306 ts_tablet_manager.cc:1939] T 4419155c2345420ea2f5ece696b5194d P c032af05a4224d24989c591c06a8294d: Deleting consensus metadata
W20260810 18:05:18.942951 15776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:41709 (error 108)
I20260810 18:05:18.943498 15776 catalog_manager.cc:4841] TS c032af05a4224d24989c591c06a8294d (127.28.83.67:38657): tablet 4419155c2345420ea2f5ece696b5194d (table test-table [id=917dc8d3b6be4764a9becb2c176fa9d6]) successfully deleted
W20260810 18:05:18.944207 16274 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:18.943832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:41709 (local address 127.0.0.1:44238)
0810 18:05:18.943973 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:18.943991 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:18.944088 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:41709: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 18:05:18.944446 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 15991
W20260810 18:05:18.944416 15776 catalog_manager.cc:4584] TS 68b0c30ce17f4835bfbbce73183c8e99 (127.28.83.65:41709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4419155c2345420ea2f5ece696b5194d: Network error: Client connection negotiation failed: client connection to 127.28.83.65:41709: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260810 18:05:18.951699 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 16128
I20260810 18:05:18.958753 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 15759
2026-08-10T18:05:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1220 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:19 dist-test-slave-srgv krb5kdc[16312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:19 dist-test-slave-srgv krb5kdc[16312](info): set up 2 sockets
Aug 10 18:05:19 dist-test-slave-srgv krb5kdc[16312](info): commencing operation
krb5kdc: starting...
W20260810 18:05:20.028971 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:20 dist-test-slave-srgv krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:20.160136 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44501
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:44501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:20.275707 16327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:20.276067 16327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:20.276151 16327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:20.280331 16327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:20.280468 16327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:20.280505 16327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:20.280544 16327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:20.280598 16327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:20.286443 16327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:44501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:20.287652 16327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:20.288734 16327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:20.298025 16333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:20.298030 16335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:20.298025 16332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:20.298151 16327 server_base.cc:1034] running on GCE node
I20260810 18:05:20.299019 16327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:20.300159 16327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:20.301363 16327 hybrid_clock.cc:648] HybridClock initialized: now 1786385120301356 us; error 32 us; skew 500 ppm
Aug 10 18:05:20 dist-test-slave-srgv krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:20.304232 16327 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:20.305109 16327 webserver.cc:460] Webserver started at http://127.28.83.126:44759/ using document root <none> and password file <none>
I20260810 18:05:20.305709 16327 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:20.305774 16327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:20.305945 16327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:20.308036 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "464f0442515f4cbbbc06d6e139ce4c22"
format_stamp: "Formatted at 2026-08-10 18:05:20 on dist-test-slave-srgv"
server_key: "dac093f5d5ef6460e77ba064bdafb0a7"
server_key_iv: "a2b612d58d197bf8f92ed605f4c2f875"
server_key_version: "encryptionkey@0"
I20260810 18:05:20.308578 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "464f0442515f4cbbbc06d6e139ce4c22"
format_stamp: "Formatted at 2026-08-10 18:05:20 on dist-test-slave-srgv"
server_key: "dac093f5d5ef6460e77ba064bdafb0a7"
server_key_iv: "a2b612d58d197bf8f92ed605f4c2f875"
server_key_version: "encryptionkey@0"
I20260810 18:05:20.312392 16327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 18:05:20.314790 16343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:20.315975 16327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:05:20.316098 16327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "464f0442515f4cbbbc06d6e139ce4c22"
format_stamp: "Formatted at 2026-08-10 18:05:20 on dist-test-slave-srgv"
server_key: "dac093f5d5ef6460e77ba064bdafb0a7"
server_key_iv: "a2b612d58d197bf8f92ed605f4c2f875"
server_key_version: "encryptionkey@0"
I20260810 18:05:20.316198 16327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:20.351286 16327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:20.354492 16327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:20.354666 16327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:20.362097 16327 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:44501
I20260810 18:05:20.362164 16399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:44501 every 8 connection(s)
I20260810 18:05:20.363128 16327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:20.366020 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:20.367715 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 16327
I20260810 18:05:20.367880 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:20.368227 29005 external_mini_cluster.cc:1428] Setting key f0eab9dfffc54e4acd518a4e97859a8d
I20260810 18:05:20.372593 16400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22: Bootstrap starting.
I20260810 18:05:20.375008 16400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:20.375912 16400 log.cc:824] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:05:20 dist-test-slave-srgv krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:20.378446 16400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22: No bootstrap required, opened a new log
I20260810 18:05:20.382918 16400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464f0442515f4cbbbc06d6e139ce4c22" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44501 } }
I20260810 18:05:20.383249 16400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:20.383343 16400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464f0442515f4cbbbc06d6e139ce4c22, State: Initialized, Role: FOLLOWER
I20260810 18:05:20.383818 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:20.369800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.28.83.126:44501)
0810 18:05:20.370291 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:20.370306 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:20.370345 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:05:20.371018 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:20.371036 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:20.371376 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:20.371746 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:20.372613 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.373710 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.374585 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.375056 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.375234 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:20.378415 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:20.378421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:20.378434 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:20.378463 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:20.380282 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:20.380986 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:20.380991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:20.380996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:20.381128 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:20.381482 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:20.381488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:20.381490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:20.381940 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:20.382112 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:20.382427 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 18:05:20.382781 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":159,"threads_started":1}
I20260810 18:05:20.383781 16400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464f0442515f4cbbbc06d6e139ce4c22" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44501 } }
I20260810 18:05:20.383992 16400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:20.384115 16400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:20.384231 16400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:20.385352 16400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464f0442515f4cbbbc06d6e139ce4c22" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44501 } }
I20260810 18:05:20.385752 16400 leader_election.cc:302] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 464f0442515f4cbbbc06d6e139ce4c22; no voters:
I20260810 18:05:20.386137 16400 leader_election.cc:288] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:20.386222 16405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:20.386461 16405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [term 1 LEADER]: Becoming Leader. State: Replica: 464f0442515f4cbbbc06d6e139ce4c22, State: Running, Role: LEADER
I20260810 18:05:20.386794 16405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464f0442515f4cbbbc06d6e139ce4c22" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44501 } }
I20260810 18:05:20.387550 16400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:20.388660 16407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "464f0442515f4cbbbc06d6e139ce4c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464f0442515f4cbbbc06d6e139ce4c22" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44501 } } }
I20260810 18:05:20.388696 16406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464f0442515f4cbbbc06d6e139ce4c22. Latest consensus state: current_term: 1 leader_uuid: "464f0442515f4cbbbc06d6e139ce4c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464f0442515f4cbbbc06d6e139ce4c22" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44501 } } }
I20260810 18:05:20.388895 16407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:20.388942 16406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:20.389504 16414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:20.392863 16414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:20.398847 16414 catalog_manager.cc:1347] Generated new cluster ID: 33c87d84376547baabdf2f9d02547290
I20260810 18:05:20.398939 16414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:20.416946 16414 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:20.418069 16414 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:20.425868 16414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22: Generated new TSK 0
I20260810 18:05:20.426635 16414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
Entry for principal kudu/127.28.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab.
I20260810 18:05:20.468547 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44501
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:20.583343 16426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:20.583706 16426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:20.583799 16426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:20.588775 16426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:20.588868 16426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:20.588964 16426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:05:20.594617 16426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.65.keytab
--principal=kudu/127.28.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:20.595783 16426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:20.596889 16426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:20.606772 16433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:20.606966 16426 server_base.cc:1034] running on GCE node
W20260810 18:05:20.606778 16435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:20.606743 16431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:20.607633 16426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:20.608350 16426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:20.609491 16426 hybrid_clock.cc:648] HybridClock initialized: now 1786385120609470 us; error 30 us; skew 500 ppm
Aug 10 18:05:20 dist-test-slave-srgv krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:20.612406 16426 init.cc:377] Logged in from keytab as kudu/127.28.83.65@KRBTEST.COM (short username kudu)
I20260810 18:05:20.613315 16426 webserver.cc:460] Webserver started at http://127.28.83.65:43699/ using document root <none> and password file <none>
I20260810 18:05:20.613935 16426 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:20.614037 16426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:20.614282 16426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:20.616405 16426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "4a5dfca210954f03929d726eff15f64b"
format_stamp: "Formatted at 2026-08-10 18:05:20 on dist-test-slave-srgv"
server_key: "d902e0cace91e645c9a9dd532ce1effc"
server_key_iv: "e138e9e439f7332ca46f6ddd07742097"
server_key_version: "encryptionkey@0"
I20260810 18:05:20.616991 16426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "4a5dfca210954f03929d726eff15f64b"
format_stamp: "Formatted at 2026-08-10 18:05:20 on dist-test-slave-srgv"
server_key: "d902e0cace91e645c9a9dd532ce1effc"
server_key_iv: "e138e9e439f7332ca46f6ddd07742097"
server_key_version: "encryptionkey@0"
I20260810 18:05:20.621127 16426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 18:05:20.623832 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:20.625123 16426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:05:20.625278 16426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "4a5dfca210954f03929d726eff15f64b"
format_stamp: "Formatted at 2026-08-10 18:05:20 on dist-test-slave-srgv"
server_key: "d902e0cace91e645c9a9dd532ce1effc"
server_key_iv: "e138e9e439f7332ca46f6ddd07742097"
server_key_version: "encryptionkey@0"
I20260810 18:05:20.625403 16426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:20.640785 16426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:20.644248 16426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:20.644503 16426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:20.645179 16426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:20.646152 16426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:20.646229 16426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:20.646299 16426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:20.646351 16426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:20.655681 16426 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:45891
I20260810 18:05:20.655702 16561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:45891 every 8 connection(s)
I20260810 18:05:20.656682 16426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
Aug 10 18:05:20 dist-test-slave-srgv krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.65@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:20.666642 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 16426
I20260810 18:05:20.666808 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:05:20.667200 29005 external_mini_cluster.cc:1428] Setting key f328cae0e4bbcc6fe383f77906cbc5d6
I20260810 18:05:20.671497 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:20.658697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:41243 (local address 127.28.83.126:44501)
0810 18:05:20.658848 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:20.658851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:20.659934 (+ 1083us) server_negotiation.cc:407] Connection header received
0810 18:05:20.660803 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:20.660806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:20.660861 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:20.660942 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:20.662207 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.662768 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.663928 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.664177 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.664264 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:20.667145 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:20.667149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:20.667152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:20.667186 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:20.669070 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:20.669715 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:20.669721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:20.669725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:20.669828 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:20.670209 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:20.670214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:20.670219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:20.670496 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:20.670616 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:20.671187 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 18:05:20.671332 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:05:20.672407 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:20.659080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44501 (local address 127.28.83.65:41243)
0810 18:05:20.659764 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:20.659808 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:20.660596 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:20.661114 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:20.661123 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:20.661544 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:20.662034 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:20.662048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.662893 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.662896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:20.663803 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:20.663813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.664326 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.664328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:20.664481 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:20.665182 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:20.665199 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:20.666949 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:20.669224 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:20.669229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:20.669244 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:20.669610 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:20.669932 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:20.669935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:20.669936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:20.670099 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:20.670614 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:20.670619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:20.670913 (+ 294us) client_negotiation.cc:769] Sending connection context
0810 18:05:20.671117 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 18:05:20.671501 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":128,"threads_started":1}
I20260810 18:05:20.673969 16562 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44501
I20260810 18:05:20.674259 16562 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:20.674808 16562 heartbeater.cc:511] Master 127.28.83.126:44501 requested a full tablet report, sending...
I20260810 18:05:20.676498 16361 ts_manager.cc:194] Registered new tserver with Master: 4a5dfca210954f03929d726eff15f64b (127.28.83.65:45891)
I20260810 18:05:20.677752 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.65@KRBTEST.COM'} at 127.28.83.65:41243
WARNING: no policy specified for kudu/127.28.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
Entry for principal kudu/127.28.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab.
I20260810 18:05:20.696941 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44501
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:20.813306 16568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:20.813700 16568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:20.813786 16568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:20.818074 16568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:20.818158 16568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:20.818251 16568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:05:20.824167 16568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.66.keytab
--principal=kudu/127.28.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:20.825388 16568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:20.826537 16568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:20.836772 16574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:20.836772 16573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:20.836912 16576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:20.837044 16568 server_base.cc:1034] running on GCE node
I20260810 18:05:20.837644 16568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:20.838347 16568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:20.839524 16568 hybrid_clock.cc:648] HybridClock initialized: now 1786385120839509 us; error 28 us; skew 500 ppm
Aug 10 18:05:20 dist-test-slave-srgv krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:20.842613 16568 init.cc:377] Logged in from keytab as kudu/127.28.83.66@KRBTEST.COM (short username kudu)
I20260810 18:05:20.843549 16568 webserver.cc:460] Webserver started at http://127.28.83.66:43757/ using document root <none> and password file <none>
I20260810 18:05:20.844115 16568 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:20.844179 16568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:20.844358 16568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:20.846513 16568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "b7d2423851fa4b01af34bc85b334801d"
format_stamp: "Formatted at 2026-08-10 18:05:20 on dist-test-slave-srgv"
server_key: "9ff3f9a5da458659ba9f9ef56573203f"
server_key_iv: "5d531a72e22b8e07221f978835f14289"
server_key_version: "encryptionkey@0"
I20260810 18:05:20.847076 16568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "b7d2423851fa4b01af34bc85b334801d"
format_stamp: "Formatted at 2026-08-10 18:05:20 on dist-test-slave-srgv"
server_key: "9ff3f9a5da458659ba9f9ef56573203f"
server_key_iv: "5d531a72e22b8e07221f978835f14289"
server_key_version: "encryptionkey@0"
I20260810 18:05:20.851114 16568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:05:20.853811 16583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:20.855099 16568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:05:20.855208 16568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "b7d2423851fa4b01af34bc85b334801d"
format_stamp: "Formatted at 2026-08-10 18:05:20 on dist-test-slave-srgv"
server_key: "9ff3f9a5da458659ba9f9ef56573203f"
server_key_iv: "5d531a72e22b8e07221f978835f14289"
server_key_version: "encryptionkey@0"
I20260810 18:05:20.855314 16568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:20.866209 16568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:20.869756 16568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:20.869931 16568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:20.870522 16568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:20.871356 16568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:20.871404 16568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:20.871446 16568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:20.871462 16568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:20.880546 16568 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:36923
I20260810 18:05:20.880672 16703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:36923 every 8 connection(s)
I20260810 18:05:20.881644 16568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:05:20.884263 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 16568
I20260810 18:05:20.884392 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:05:20.884673 29005 external_mini_cluster.cc:1428] Setting key b5d9d38ff06fac7390b5b4df4f590a15
Aug 10 18:05:20 dist-test-slave-srgv krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385120, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.66@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.67@KRBTEST.COM" created.
I20260810 18:05:20.898330 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:20.883639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:51957 (local address 127.28.83.126:44501)
0810 18:05:20.883778 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:20.883782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:20.885096 (+ 1314us) server_negotiation.cc:407] Connection header received
0810 18:05:20.886158 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:20.886160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:20.886201 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:20.886310 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:20.887500 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.888431 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.890318 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.890540 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.890624 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:20.894064 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:20.894068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:20.894070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:20.894098 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:20.895682 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:20.896320 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:20.896323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:20.896325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:20.896405 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:20.896896 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:20.896900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:20.896901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:20.897123 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:20.897235 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:20.897985 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 18:05:20.898158 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:05:20.899514 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:20.884073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44501 (local address 127.28.83.66:51957)
0810 18:05:20.884911 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:20.884971 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:20.885955 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:20.886472 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:20.886479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:20.886862 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:20.887356 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:20.887368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.888578 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.888584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:20.890170 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:20.890186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:20.890723 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:20.890730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:20.890991 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:20.891765 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:20.891790 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:20.893846 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:20.895868 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:20.895875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:20.895901 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:20.896191 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:20.896506 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:20.896510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:20.896512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:20.896787 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:20.897251 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:20.897260 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:20.897719 (+ 459us) client_negotiation.cc:769] Sending connection context
0810 18:05:20.897926 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 18:05:20.898354 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":152,"threads_started":1}
I20260810 18:05:20.900962 16704 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44501
I20260810 18:05:20.901243 16704 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:20.901815 16704 heartbeater.cc:511] Master 127.28.83.126:44501 requested a full tablet report, sending...
I20260810 18:05:20.903012 16361 ts_manager.cc:194] Registered new tserver with Master: b7d2423851fa4b01af34bc85b334801d (127.28.83.66:36923)
I20260810 18:05:20.903928 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.66@KRBTEST.COM'} at 127.28.83.66:51957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
Entry for principal kudu/127.28.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab.
I20260810 18:05:20.914824 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44501
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:36969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:21.030886 16709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:21.031190 16709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:21.031237 16709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:21.035372 16709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:21.035444 16709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:21.035506 16709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:05:21.040956 16709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:36969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.67.keytab
--principal=kudu/127.28.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:21.042098 16709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:21.043100 16709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:21.053341 16715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:21.053344 16718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:21.053341 16716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:21.053383 16709 server_base.cc:1034] running on GCE node
I20260810 18:05:21.054111 16709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:21.054798 16709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:21.055966 16709 hybrid_clock.cc:648] HybridClock initialized: now 1786385121055951 us; error 30 us; skew 500 ppm
Aug 10 18:05:21 dist-test-slave-srgv krb5kdc[16312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385121, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:21.058990 16709 init.cc:377] Logged in from keytab as kudu/127.28.83.67@KRBTEST.COM (short username kudu)
I20260810 18:05:21.060014 16709 webserver.cc:460] Webserver started at http://127.28.83.67:39539/ using document root <none> and password file <none>
I20260810 18:05:21.060606 16709 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:21.060680 16709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:21.060926 16709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:21.063215 16709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "7e9615f1f5d544c7aa6d5ba1e8013616"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "91c0bfefad256be6d22b318e373b46fc"
server_key_iv: "b61b0b17eda53cf5b6b12aa53d275d4f"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.063828 16709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "7e9615f1f5d544c7aa6d5ba1e8013616"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "91c0bfefad256be6d22b318e373b46fc"
server_key_iv: "b61b0b17eda53cf5b6b12aa53d275d4f"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.067847 16709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 18:05:21.070703 16725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.072086 16709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:05:21.072255 16709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "7e9615f1f5d544c7aa6d5ba1e8013616"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "91c0bfefad256be6d22b318e373b46fc"
server_key_iv: "b61b0b17eda53cf5b6b12aa53d275d4f"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.072386 16709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:21.093371 16709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:21.096686 16709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:21.096917 16709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:21.097599 16709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:21.098519 16709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:21.098594 16709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.098670 16709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:21.098712 16709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.107954 16709 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:37051
I20260810 18:05:21.107983 16845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:37051 every 8 connection(s)
I20260810 18:05:21.108909 16709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:05:21.112481 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 16709
I20260810 18:05:21.112651 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:05:21.112969 29005 external_mini_cluster.cc:1428] Setting key bbea95c5870f41ccf8011ba41d116cd6
Aug 10 18:05:21 dist-test-slave-srgv krb5kdc[16312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385121, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.67@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:21.124109 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.110878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:36417 (local address 127.28.83.126:44501)
0810 18:05:21.111055 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.111060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.112188 (+ 1128us) server_negotiation.cc:407] Connection header received
0810 18:05:21.113446 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.113450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.113515 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.113631 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:21.114989 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.115523 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.116703 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.116925 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.117038 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.120026 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:21.120032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:21.120059 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:21.120096 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:21.121842 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:21.122424 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:21.122429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:21.122431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:21.122518 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:21.122908 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:21.122913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:21.122914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:21.123118 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:21.123253 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.123781 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.123914 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 18:05:21.125001 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.111295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44501 (local address 127.28.83.67:36417)
0810 18:05:21.111973 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:21.112017 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:21.113167 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:21.113808 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:21.113815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:21.114343 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:21.114842 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.114856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.115673 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.115677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.116576 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.116587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.117092 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.117101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.117279 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.118012 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:21.118029 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:21.119806 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:21.121978 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:21.121985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:21.122003 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:21.122308 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:21.122638 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:21.122641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:21.122643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:21.122804 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:21.123235 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:21.123241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:21.123529 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 18:05:21.123740 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 18:05:21.124131 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":122,"threads_started":1}
I20260810 18:05:21.126365 16846 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44501
I20260810 18:05:21.126672 16846 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:21.127264 16846 heartbeater.cc:511] Master 127.28.83.126:44501 requested a full tablet report, sending...
I20260810 18:05:21.128352 16361 ts_manager.cc:194] Registered new tserver with Master: 7e9615f1f5d544c7aa6d5ba1e8013616 (127.28.83.67:37051)
I20260810 18:05:21.128888 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.83.67@KRBTEST.COM'} at 127.28.83.67:36417
I20260810 18:05:21.139274 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:05:21.149232 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.141334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.28.83.126:44501)
0810 18:05:21.141512 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.141516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.141729 (+ 213us) server_negotiation.cc:407] Connection header received
0810 18:05:21.141900 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.141903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.141952 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.142036 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:21.142796 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.143371 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.144266 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.144493 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.144608 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.146129 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:21.146132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:21.146134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:21.146159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:21.147460 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:21.147920 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:21.147929 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:21.147930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:21.148007 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:21.148289 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:21.148293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:21.148295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:21.148532 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:21.148683 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.148930 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.149051 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 18:05:21.152308 16361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:05:21.155392 16361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:05:21.170600 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.164075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:45891 (local address 127.0.0.1:59896)
0810 18:05:21.164992 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:21.165009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:21.165112 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:21.165783 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:21.165788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:21.165809 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:21.165983 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.166068 (+ 85us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.167552 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.167555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.169032 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.169045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.169990 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.169993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.170124 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.170169 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 18:05:21.170281 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 18:05:21.170419 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":87,"thread_start_us":105,"threads_started":1}
I20260810 18:05:21.170833 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.164287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:37051 (local address 127.0.0.1:33178)
0810 18:05:21.165472 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:21.165486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:21.165563 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:21.166578 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:21.166581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:21.166599 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:21.166689 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.166694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.167951 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.167956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.169080 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.169090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.170413 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.170420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.170540 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.170565 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:05:21.170637 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:05:21.170709 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1103,"mutex_wait_us":57,"thread_start_us":121,"threads_started":1}
I20260810 18:05:21.170964 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.164308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.28.83.65:45891)
0810 18:05:21.164733 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.164752 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.165248 (+ 496us) server_negotiation.cc:407] Connection header received
0810 18:05:21.165338 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.165343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.165596 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.165775 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:21.166185 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.167414 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.169179 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.169862 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.169989 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.170153 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.170687 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.170806 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":121,"threads_started":1}
I20260810 18:05:21.171293 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.164871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.28.83.67:37051)
0810 18:05:21.166026 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.166045 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.166068 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:05:21.166166 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.166170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.166415 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.166566 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:21.166813 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.167819 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.169215 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.170081 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.170183 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.170316 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.171015 (+ 699us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.171135 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":163,"threads_started":1}
I20260810 18:05:21.171335 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.164886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.28.83.66:36923)
0810 18:05:21.165395 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.165430 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.165454 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:05:21.165535 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.165541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.165767 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.166199 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:21.166333 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.167207 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.169200 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.170071 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.170183 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.170317 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.170471 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.170586 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":161,"threads_started":1}
I20260810 18:05:21.170600 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.164024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:36923 (local address 127.0.0.1:52588)
0810 18:05:21.164883 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:21.164937 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:21.165092 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:21.165912 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:21.165915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:21.165928 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:21.166022 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.166028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.167336 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.167341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.169067 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.169076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.170233 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.170240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.170332 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.170350 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:05:21.170422 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:05:21.170496 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":191,"thread_start_us":108,"threads_started":1}
I20260810 18:05:21.174414 16493 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ce74e8e1547bcb0be45c476023d8c (DEFAULT_TABLE table=test-table [id=3713125528114ff191464bc557c89257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:21.174417 16635 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ce74e8e1547bcb0be45c476023d8c (DEFAULT_TABLE table=test-table [id=3713125528114ff191464bc557c89257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:21.175513 16493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3ce74e8e1547bcb0be45c476023d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:21.175503 16635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3ce74e8e1547bcb0be45c476023d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:21.182291 16863 tablet_bootstrap.cc:492] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b: Bootstrap starting.
I20260810 18:05:21.182936 16775 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ce74e8e1547bcb0be45c476023d8c (DEFAULT_TABLE table=test-table [id=3713125528114ff191464bc557c89257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:21.183425 16864 tablet_bootstrap.cc:492] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: Bootstrap starting.
I20260810 18:05:21.184201 16775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3ce74e8e1547bcb0be45c476023d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:21.184803 16863 tablet_bootstrap.cc:654] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:21.185588 16864 tablet_bootstrap.cc:654] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:21.185688 16863 log.cc:824] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:21.186892 16864 log.cc:824] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:21.189177 16864 tablet_bootstrap.cc:492] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: No bootstrap required, opened a new log
I20260810 18:05:21.189395 16864 ts_tablet_manager.cc:1397] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20260810 18:05:21.189554 16863 tablet_bootstrap.cc:492] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b: No bootstrap required, opened a new log
I20260810 18:05:21.189862 16863 ts_tablet_manager.cc:1397] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:05:21.192665 16867 tablet_bootstrap.cc:492] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: Bootstrap starting.
I20260810 18:05:21.193742 16864 raft_consensus.cc:348] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } }
I20260810 18:05:21.194239 16864 raft_consensus.cc:374] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:21.194350 16864 raft_consensus.cc:729] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d2423851fa4b01af34bc85b334801d, State: Initialized, Role: FOLLOWER
I20260810 18:05:21.194414 16863 raft_consensus.cc:348] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } }
I20260810 18:05:21.194721 16863 raft_consensus.cc:374] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:21.194787 16863 raft_consensus.cc:729] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5dfca210954f03929d726eff15f64b, State: Initialized, Role: FOLLOWER
I20260810 18:05:21.195041 16864 consensus_queue.cc:260] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } }
I20260810 18:05:21.195264 16863 consensus_queue.cc:260] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } }
I20260810 18:05:21.195734 16867 tablet_bootstrap.cc:654] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:21.195883 16704 heartbeater.cc:503] Master 127.28.83.126:44501 was elected leader, sending a full tablet report...
I20260810 18:05:21.196218 16864 ts_tablet_manager.cc:1428] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260810 18:05:21.196704 16867 log.cc:824] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:21.197516 16562 heartbeater.cc:503] Master 127.28.83.126:44501 was elected leader, sending a full tablet report...
I20260810 18:05:21.198608 16863 ts_tablet_manager.cc:1428] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260810 18:05:21.199172 16867 tablet_bootstrap.cc:492] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: No bootstrap required, opened a new log
I20260810 18:05:21.199404 16867 ts_tablet_manager.cc:1397] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:05:21.203178 16867 raft_consensus.cc:348] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } }
I20260810 18:05:21.203517 16867 raft_consensus.cc:374] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:21.203606 16867 raft_consensus.cc:729] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9615f1f5d544c7aa6d5ba1e8013616, State: Initialized, Role: FOLLOWER
I20260810 18:05:21.204108 16867 consensus_queue.cc:260] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } }
I20260810 18:05:21.204851 16846 heartbeater.cc:503] Master 127.28.83.126:44501 was elected leader, sending a full tablet report...
I20260810 18:05:21.205107 16867 ts_tablet_manager.cc:1428] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:05:21.215826 16869 raft_consensus.cc:482] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:21.216095 16869 raft_consensus.cc:504] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } }
I20260810 18:05:21.217303 16869 leader_election.cc:288] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7d2423851fa4b01af34bc85b334801d (127.28.83.66:36923), 7e9615f1f5d544c7aa6d5ba1e8013616 (127.28.83.67:37051)
I20260810 18:05:21.221658 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.217818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:40197 (local address 127.28.83.67:37051)
0810 18:05:21.217960 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.217965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.218259 (+ 294us) server_negotiation.cc:407] Connection header received
0810 18:05:21.218414 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.218418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.218470 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.218559 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:21.218858 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.219368 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.220604 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.221025 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.221164 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.221221 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.221397 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.221508 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:05:21.221688 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.217608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:37051 (local address 127.28.83.65:40197)
0810 18:05:21.218153 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:21.218165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:21.218306 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:21.218573 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:21.218576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:21.218591 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:21.218742 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.218748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.219500 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.219503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.220431 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.220441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.221163 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.221166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.221265 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.221290 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:05:21.221379 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 18:05:21.221488 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":88,"thread_start_us":113,"threads_started":1}
I20260810 18:05:21.221679 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.217735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:38429 (local address 127.28.83.66:36923)
0810 18:05:21.217881 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.217885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.218184 (+ 299us) server_negotiation.cc:407] Connection header received
0810 18:05:21.218352 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.218355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.218410 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.218501 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:21.218799 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.219311 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.220623 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.221061 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.221207 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.221266 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.221471 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.221552 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:05:21.221712 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.217608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:36923 (local address 127.28.83.65:38429)
0810 18:05:21.218049 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:21.218062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:21.218240 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:21.218512 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:21.218514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:21.218531 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:21.218681 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.218688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.219478 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.219484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.220487 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.220517 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.221204 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.221208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.221316 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.221345 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:05:21.221454 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 18:05:21.221531 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":28,"thread_start_us":153,"threads_started":1}
I20260810 18:05:21.222620 16656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ce74e8e1547bcb0be45c476023d8c" candidate_uuid: "4a5dfca210954f03929d726eff15f64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d2423851fa4b01af34bc85b334801d" is_pre_election: true
I20260810 18:05:21.222658 16797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ce74e8e1547bcb0be45c476023d8c" candidate_uuid: "4a5dfca210954f03929d726eff15f64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" is_pre_election: true
I20260810 18:05:21.222958 16797 raft_consensus.cc:2454] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5dfca210954f03929d726eff15f64b in term 0.
I20260810 18:05:21.222965 16656 raft_consensus.cc:2454] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5dfca210954f03929d726eff15f64b in term 0.
I20260810 18:05:21.223482 16446 leader_election.cc:302] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a5dfca210954f03929d726eff15f64b, 7e9615f1f5d544c7aa6d5ba1e8013616; no voters:
I20260810 18:05:21.223816 16869 raft_consensus.cc:2781] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:05:21.223899 16869 raft_consensus.cc:482] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:05:21.223932 16869 raft_consensus.cc:3037] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:21.225102 16869 raft_consensus.cc:504] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } }
I20260810 18:05:21.225584 16869 leader_election.cc:288] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [CANDIDATE]: Term 1 election: Requested vote from peers b7d2423851fa4b01af34bc85b334801d (127.28.83.66:36923), 7e9615f1f5d544c7aa6d5ba1e8013616 (127.28.83.67:37051)
I20260810 18:05:21.225975 16656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ce74e8e1547bcb0be45c476023d8c" candidate_uuid: "4a5dfca210954f03929d726eff15f64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d2423851fa4b01af34bc85b334801d"
I20260810 18:05:21.226120 16656 raft_consensus.cc:3037] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:21.226176 16797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ce74e8e1547bcb0be45c476023d8c" candidate_uuid: "4a5dfca210954f03929d726eff15f64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616"
I20260810 18:05:21.226281 16797 raft_consensus.cc:3037] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:21.227397 16656 raft_consensus.cc:2454] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5dfca210954f03929d726eff15f64b in term 1.
I20260810 18:05:21.227406 16797 raft_consensus.cc:2454] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5dfca210954f03929d726eff15f64b in term 1.
I20260810 18:05:21.227766 16443 leader_election.cc:302] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a5dfca210954f03929d726eff15f64b, b7d2423851fa4b01af34bc85b334801d; no voters:
I20260810 18:05:21.227957 16869 raft_consensus.cc:2781] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:21.228202 16869 raft_consensus.cc:686] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [term 1 LEADER]: Becoming Leader. State: Replica: 4a5dfca210954f03929d726eff15f64b, State: Running, Role: LEADER
I20260810 18:05:21.228617 16869 consensus_queue.cc:237] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } }
I20260810 18:05:21.232371 16361 catalog_manager.cc:5495] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a5dfca210954f03929d726eff15f64b (127.28.83.65). New cstate: current_term: 1 leader_uuid: "4a5dfca210954f03929d726eff15f64b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a5dfca210954f03929d726eff15f64b" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 45891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 } health_report { overall_health: UNKNOWN } } }
I20260810 18:05:21.251096 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.247388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.28.83.65:45891)
0810 18:05:21.247539 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.247544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.247560 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:05:21.247678 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.247681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.247743 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.247872 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:21.248126 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.248639 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.249598 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.249816 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.249925 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.250265 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:21.250687 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:21.250818 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.250878 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.250942 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:05:21.257326 16797 raft_consensus.cc:1261] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [term 1 FOLLOWER]: Refusing update from remote peer 4a5dfca210954f03929d726eff15f64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:21.257336 16656 raft_consensus.cc:1261] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [term 1 FOLLOWER]: Refusing update from remote peer 4a5dfca210954f03929d726eff15f64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:21.258136 16869 consensus_queue.cc:1035] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9615f1f5d544c7aa6d5ba1e8013616" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:21.258373 16874 consensus_queue.cc:1035] T 2f3ce74e8e1547bcb0be45c476023d8c P 4a5dfca210954f03929d726eff15f64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d2423851fa4b01af34bc85b334801d" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:21.266498 16877 mvcc.cc:205] Tried to move back new op lower bound from 7317033456662274048 to 7317033456555765760. Current Snapshot: MvccSnapshot[applied={T|T < 7317033456662274048}]
I20260810 18:05:21.267127 16879 mvcc.cc:205] Tried to move back new op lower bound from 7317033456662274048 to 7317033456555765760. Current Snapshot: MvccSnapshot[applied={T|T < 7317033456662274048}]
I20260810 18:05:21.268114 16878 mvcc.cc:205] Tried to move back new op lower bound from 7317033456662274048 to 7317033456555765760. Current Snapshot: MvccSnapshot[applied={T|T < 7317033456662274048}]
I20260810 18:05:21.277796 16360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:21.278016 16360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:21.280741 16360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 464f0442515f4cbbbc06d6e139ce4c22: Sending DeleteTablet for 3 replicas of tablet 2f3ce74e8e1547bcb0be45c476023d8c
I20260810 18:05:21.281652 16493 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3ce74e8e1547bcb0be45c476023d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:21 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:59896
I20260810 18:05:21.281672 16635 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3ce74e8e1547bcb0be45c476023d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:21 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:52588
I20260810 18:05:21.281736 16775 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3ce74e8e1547bcb0be45c476023d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:21 UTC) from {username='kudu', principal='kudu/127.28.83.126@KRBTEST.COM'} at 127.0.0.1:33178
I20260810 18:05:21.282351 16891 tablet_replica.cc:330] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: stopping tablet replica
I20260810 18:05:21.282729 16891 raft_consensus.cc:2229] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:21.283217 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 16426
I20260810 18:05:21.285717 16891 raft_consensus.cc:2258] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:21.285902 16889 tablet_replica.cc:330] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: stopping tablet replica
I20260810 18:05:21.286255 16889 raft_consensus.cc:2229] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:21.286623 16889 raft_consensus.cc:2258] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:21.287925 16891 ts_tablet_manager.cc:1905] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:21.288059 16889 ts_tablet_manager.cc:1905] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:21.290799 16889 ts_tablet_manager.cc:1918] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:21.290913 16889 log.cc:1196] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/2f3ce74e8e1547bcb0be45c476023d8c
I20260810 18:05:21.291324 16889 ts_tablet_manager.cc:1939] T 2f3ce74e8e1547bcb0be45c476023d8c P b7d2423851fa4b01af34bc85b334801d: Deleting consensus metadata
I20260810 18:05:21.292608 16344 catalog_manager.cc:4841] TS b7d2423851fa4b01af34bc85b334801d (127.28.83.66:36923): tablet 2f3ce74e8e1547bcb0be45c476023d8c (table test-table [id=3713125528114ff191464bc557c89257]) successfully deleted
I20260810 18:05:21.292949 16891 ts_tablet_manager.cc:1918] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:21.293063 16891 log.cc:1196] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/2f3ce74e8e1547bcb0be45c476023d8c
W20260810 18:05:21.293255 16346 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:45891 (error 108)
I20260810 18:05:21.293408 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 16568
I20260810 18:05:21.293443 16891 ts_tablet_manager.cc:1939] T 2f3ce74e8e1547bcb0be45c476023d8c P 7e9615f1f5d544c7aa6d5ba1e8013616: Deleting consensus metadata
I20260810 18:05:21.294708 16347 catalog_manager.cc:4841] TS 7e9615f1f5d544c7aa6d5ba1e8013616 (127.28.83.67:37051): tablet 2f3ce74e8e1547bcb0be45c476023d8c (table test-table [id=3713125528114ff191464bc557c89257]) successfully deleted
I20260810 18:05:21.295475 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.295051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:45891 (local address 127.0.0.1:59908)
0810 18:05:21.295272 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:21.295365 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260810 18:05:21.295653 16346 catalog_manager.cc:4584] TS 4a5dfca210954f03929d726eff15f64b (127.28.83.65:45891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f3ce74e8e1547bcb0be45c476023d8c: Network error: Client connection negotiation failed: client connection to 127.28.83.65:45891: connect: Connection refused (error 111)
I20260810 18:05:21.300732 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 16709
I20260810 18:05:21.307353 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 16327
2026-08-10T18:05:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2349 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T18:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:21Z Disabled control of system clock
I20260810 18:05:21.340129 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44197
--webserver_interface=127.28.83.126
--webserver_port=0
--builtin_ntp_servers=127.28.83.84:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:44197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:05:21.456046 16897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:21.456351 16897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:21.456393 16897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:21.460521 16897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:21.460601 16897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:21.460618 16897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:21.460635 16897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:21.460651 16897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:21.466365 16897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:44197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:44197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:21.467528 16897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:21.468973 16897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:21.477689 16905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:21.477648 16902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:21.477664 16903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:21.478003 16897 server_base.cc:1034] running on GCE node
I20260810 18:05:21.478663 16897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:21.479928 16897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:21.481204 16897 hybrid_clock.cc:648] HybridClock initialized: now 1786385121481173 us; error 60 us; skew 500 ppm
I20260810 18:05:21.483371 16897 webserver.cc:460] Webserver started at http://127.28.83.126:42485/ using document root <none> and password file <none>
I20260810 18:05:21.484005 16897 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:21.484138 16897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:21.484395 16897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:21.486800 16897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "3d44e25cce8b46aaa8412a9150f5e964"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "f2a3c7092e42988259e42178ef46b5c7"
server_key_iv: "1224f67a9f8d737b0c7e9f367c768550"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.487455 16897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "3d44e25cce8b46aaa8412a9150f5e964"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "f2a3c7092e42988259e42178ef46b5c7"
server_key_iv: "1224f67a9f8d737b0c7e9f367c768550"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.491587 16897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:21.494220 16911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.495538 16897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:05:21.495714 16897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "3d44e25cce8b46aaa8412a9150f5e964"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "f2a3c7092e42988259e42178ef46b5c7"
server_key_iv: "1224f67a9f8d737b0c7e9f367c768550"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.495847 16897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:21.510100 16897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:21.510890 16897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:21.511128 16897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:21.518674 16897 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:44197
I20260810 18:05:21.518700 16966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:44197 every 8 connection(s)
I20260810 18:05:21.519704 16897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:21.523058 16967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:21.526973 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 16897
I20260810 18:05:21.527091 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:21.527388 29005 external_mini_cluster.cc:1428] Setting key d889ed230468b2a873ce0b52c56c9fed
I20260810 18:05:21.529207 16967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964: Bootstrap starting.
I20260810 18:05:21.531806 16967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:21.532626 16967 log.cc:824] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:21.535022 16967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964: No bootstrap required, opened a new log
I20260810 18:05:21.536996 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.528922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.28.83.126:44197)
0810 18:05:21.529643 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.529660 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.529697 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:05:21.530380 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.530397 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.530801 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.531090 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:21.531459 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.532514 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.533684 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.534152 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.534300 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.535198 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:21.535216 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:21.535229 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:21.535627 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:21.535684 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:21.535704 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:21.535780 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.535957 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.536275 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":148,"threads_started":1}
I20260810 18:05:21.539158 16967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d44e25cce8b46aaa8412a9150f5e964" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44197 } }
I20260810 18:05:21.539474 16967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:21.539541 16967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d44e25cce8b46aaa8412a9150f5e964, State: Initialized, Role: FOLLOWER
I20260810 18:05:21.539950 16967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d44e25cce8b46aaa8412a9150f5e964" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44197 } }
I20260810 18:05:21.540067 16967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:21.540112 16967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:21.540182 16967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:21.541175 16967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d44e25cce8b46aaa8412a9150f5e964" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44197 } }
I20260810 18:05:21.541544 16967 leader_election.cc:302] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d44e25cce8b46aaa8412a9150f5e964; no voters:
I20260810 18:05:21.541887 16967 leader_election.cc:288] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:21.542001 16973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:21.542243 16973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [term 1 LEADER]: Becoming Leader. State: Replica: 3d44e25cce8b46aaa8412a9150f5e964, State: Running, Role: LEADER
I20260810 18:05:21.542598 16973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d44e25cce8b46aaa8412a9150f5e964" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44197 } }
I20260810 18:05:21.543210 16967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:21.544304 16974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d44e25cce8b46aaa8412a9150f5e964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d44e25cce8b46aaa8412a9150f5e964" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44197 } } }
I20260810 18:05:21.544471 16974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:21.544826 16975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d44e25cce8b46aaa8412a9150f5e964. Latest consensus state: current_term: 1 leader_uuid: "3d44e25cce8b46aaa8412a9150f5e964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d44e25cce8b46aaa8412a9150f5e964" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 44197 } } }
I20260810 18:05:21.544945 16975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:21.544880 16982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:21.547796 16982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:21.554133 16982 catalog_manager.cc:1347] Generated new cluster ID: 48aefb23470d410ab8be914a45dc0be8
I20260810 18:05:21.554234 16982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:21.564204 16982 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:21.565186 16982 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:21.574254 16982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964: Generated new TSK 0
I20260810 18:05:21.575152 16982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:21.582780 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.65:0
--local_ip_for_outbound_sockets=127.28.83.65
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44197
--builtin_ntp_servers=127.28.83.84:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:05:21.701915 16992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:21.702247 16992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:21.702350 16992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:21.706576 16992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:21.706689 16992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:21.706795 16992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.65
I20260810 18:05:21.711957 16992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.83.65
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.65
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:21.713181 16992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:21.714707 16992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:21.725087 16997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:21.725183 16998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:21.725314 16992 server_base.cc:1034] running on GCE node
W20260810 18:05:21.725087 17000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:21.725993 16992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:21.726730 16992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:21.727942 16992 hybrid_clock.cc:648] HybridClock initialized: now 1786385121727927 us; error 26 us; skew 500 ppm
I20260810 18:05:21.730397 16992 webserver.cc:460] Webserver started at http://127.28.83.65:32965/ using document root <none> and password file <none>
I20260810 18:05:21.731052 16992 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:21.731156 16992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:21.731403 16992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:21.733773 16992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/data/instance:
uuid: "c634b5450c9b4b8d82311be9276de9b4"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "ef0f866bdb1dc00bc0121afb35869a6d"
server_key_iv: "19a9aa69ce6b7691f9cbd8cfc59c36c6"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.734391 16992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance:
uuid: "c634b5450c9b4b8d82311be9276de9b4"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "ef0f866bdb1dc00bc0121afb35869a6d"
server_key_iv: "19a9aa69ce6b7691f9cbd8cfc59c36c6"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.738436 16992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:05:21.741247 17006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.742606 16992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:21.742775 16992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/wal
uuid: "c634b5450c9b4b8d82311be9276de9b4"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "ef0f866bdb1dc00bc0121afb35869a6d"
server_key_iv: "19a9aa69ce6b7691f9cbd8cfc59c36c6"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.742910 16992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:21.758582 16992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:21.759343 16992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:21.759555 16992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:21.760325 16992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:21.761323 16992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:21.761399 16992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.761499 16992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:21.761554 16992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.770869 16992 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.65:42269
I20260810 18:05:21.770892 17125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.65:42269 every 8 connection(s)
I20260810 18:05:21.771926 16992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/data/info.pb
I20260810 18:05:21.780776 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 16992
I20260810 18:05:21.780913 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-0/wal/instance
I20260810 18:05:21.781273 29005 external_mini_cluster.cc:1428] Setting key c525ac41f137ea21ea3830d11facb047
I20260810 18:05:21.783623 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.775294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.65:59897 (local address 127.28.83.126:44197)
0810 18:05:21.775477 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.775481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.776581 (+ 1100us) server_negotiation.cc:407] Connection header received
0810 18:05:21.778163 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.778166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.778231 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.778346 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:21.779312 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.779881 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.781281 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.781628 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.781755 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.783083 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:21.783087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:21.783092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:21.783110 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:21.783131 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:21.783139 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:21.783221 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.783397 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.783466 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 18:05:21.783667 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.66:0
--local_ip_for_outbound_sockets=127.28.83.66
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44197
--builtin_ntp_servers=127.28.83.84:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:05:21.784660 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.775576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44197 (local address 127.28.83.65:59897)
0810 18:05:21.776396 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:21.776441 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:21.777920 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:21.778532 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:21.778538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:21.778634 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:21.779158 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.779170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.780047 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.780050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.781135 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.781146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.781809 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.781814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.781996 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.782789 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:05:21.782791 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:21.782837 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:05:21.782913 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:21.783225 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:21.783238 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:21.783245 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:05:21.783387 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 18:05:21.783800 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":193,"threads_started":1}
I20260810 18:05:21.786144 17126 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44197
I20260810 18:05:21.786418 17126 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:21.787005 17126 heartbeater.cc:511] Master 127.28.83.126:44197 requested a full tablet report, sending...
I20260810 18:05:21.788694 16928 ts_manager.cc:194] Registered new tserver with Master: c634b5450c9b4b8d82311be9276de9b4 (127.28.83.65:42269)
I20260810 18:05:21.789970 16928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.65:59897
W20260810 18:05:21.904091 17129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:21.904448 17129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:21.904525 17129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:21.908747 17129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:21.908867 17129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:21.908973 17129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.66
I20260810 18:05:21.914250 17129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.83.66
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.66
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:21.915460 17129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:21.916913 17129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:21.926780 17137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:21.926770 17134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:21.926774 17135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:21.927202 17129 server_base.cc:1034] running on GCE node
I20260810 18:05:21.927587 17129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:21.928254 17129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:21.929492 17129 hybrid_clock.cc:648] HybridClock initialized: now 1786385121929477 us; error 29 us; skew 500 ppm
I20260810 18:05:21.931442 17129 webserver.cc:460] Webserver started at http://127.28.83.66:35045/ using document root <none> and password file <none>
I20260810 18:05:21.932058 17129 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:21.932157 17129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:21.932408 17129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:21.934608 17129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/data/instance:
uuid: "4138bdb316c748dba586c619481e9fd2"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "978d228277a03d61b64c6f6af49f4819"
server_key_iv: "5547c2e2ef4c5863d527478abb4be1ae"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.935231 17129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance:
uuid: "4138bdb316c748dba586c619481e9fd2"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "978d228277a03d61b64c6f6af49f4819"
server_key_iv: "5547c2e2ef4c5863d527478abb4be1ae"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.939154 17129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:21.941788 17144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.943116 17129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:05:21.943274 17129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/wal
uuid: "4138bdb316c748dba586c619481e9fd2"
format_stamp: "Formatted at 2026-08-10 18:05:21 on dist-test-slave-srgv"
server_key: "978d228277a03d61b64c6f6af49f4819"
server_key_iv: "5547c2e2ef4c5863d527478abb4be1ae"
server_key_version: "encryptionkey@0"
I20260810 18:05:21.943416 17129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:21.966199 17129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:21.966912 17129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:21.967085 17129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:21.967643 17129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:21.968603 17129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:21.968662 17129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.968704 17129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:21.968719 17129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:21.977779 17129 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.66:41097
I20260810 18:05:21.977806 17263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.66:41097 every 8 connection(s)
I20260810 18:05:21.978760 17129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/data/info.pb
I20260810 18:05:21.982211 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 17129
I20260810 18:05:21.982381 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/wal/instance
I20260810 18:05:21.982726 29005 external_mini_cluster.cc:1428] Setting key bda708a85d8a174b9c664540deb56233
I20260810 18:05:21.985008 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.83.67:0
--local_ip_for_outbound_sockets=127.28.83.67
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44197
--builtin_ntp_servers=127.28.83.84:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:05:21.989912 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.980744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.66:46619 (local address 127.28.83.126:44197)
0810 18:05:21.980897 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:21.980901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:21.982217 (+ 1316us) server_negotiation.cc:407] Connection header received
0810 18:05:21.984291 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:21.984294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:21.984337 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:21.984439 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:21.985450 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.986288 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.987533 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.987916 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.988062 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.989283 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:21.989289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:21.989295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:21.989315 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:21.989344 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:21.989358 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:21.989489 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:21.989636 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 18:05:21.989718 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:05:21.990805 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:21.981191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44197 (local address 127.28.83.66:46619)
0810 18:05:21.982018 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:21.982076 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:21.984009 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:21.984618 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:21.984624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:21.984709 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:21.985275 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.985290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.986442 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.986447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.987403 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:21.987412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:21.988078 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:21.988080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:21.988314 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:21.989002 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:05:21.989004 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:21.989047 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:05:21.989116 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:21.989490 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:21.989499 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:21.989503 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:05:21.989605 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 18:05:21.989935 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":193,"threads_started":1}
I20260810 18:05:21.992038 17264 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44197
I20260810 18:05:21.992272 17264 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:21.992817 17264 heartbeater.cc:511] Master 127.28.83.126:44197 requested a full tablet report, sending...
I20260810 18:05:21.993947 16928 ts_manager.cc:194] Registered new tserver with Master: 4138bdb316c748dba586c619481e9fd2 (127.28.83.66:41097)
I20260810 18:05:21.994560 16928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.66:46619
W20260810 18:05:22.107784 17268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:22.108075 17268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:22.108117 17268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:22.112377 17268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:22.112452 17268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:22.112514 17268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.83.67
I20260810 18:05:22.117580 17268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.83.67
--webserver_port=0
--tserver_master_addrs=127.28.83.126:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.83.67
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:22.118781 17268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:22.120210 17268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:22.130081 17274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:22.130177 17268 server_base.cc:1034] running on GCE node
W20260810 18:05:22.130121 17276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:22.130122 17273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:22.130777 17268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:22.131460 17268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:22.132648 17268 hybrid_clock.cc:648] HybridClock initialized: now 1786385122132618 us; error 52 us; skew 500 ppm
I20260810 18:05:22.134572 17268 webserver.cc:460] Webserver started at http://127.28.83.67:45491/ using document root <none> and password file <none>
I20260810 18:05:22.135183 17268 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:22.135280 17268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:22.135524 17268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:22.137748 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/data/instance:
uuid: "dc84a87d3ede4edea800dca92b091d8a"
format_stamp: "Formatted at 2026-08-10 18:05:22 on dist-test-slave-srgv"
server_key: "a60cab51119d2f4dbd5e2293b269e745"
server_key_iv: "997e26899d6d3bec0eb9e20a167df056"
server_key_version: "encryptionkey@0"
I20260810 18:05:22.138342 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance:
uuid: "dc84a87d3ede4edea800dca92b091d8a"
format_stamp: "Formatted at 2026-08-10 18:05:22 on dist-test-slave-srgv"
server_key: "a60cab51119d2f4dbd5e2293b269e745"
server_key_iv: "997e26899d6d3bec0eb9e20a167df056"
server_key_version: "encryptionkey@0"
I20260810 18:05:22.142204 17268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:05:22.144889 17283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:22.146153 17268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 18:05:22.146294 17268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/wal
uuid: "dc84a87d3ede4edea800dca92b091d8a"
format_stamp: "Formatted at 2026-08-10 18:05:22 on dist-test-slave-srgv"
server_key: "a60cab51119d2f4dbd5e2293b269e745"
server_key_iv: "997e26899d6d3bec0eb9e20a167df056"
server_key_version: "encryptionkey@0"
I20260810 18:05:22.146418 17268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:22.175735 17268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:22.176533 17268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:22.176750 17268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:22.177383 17268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:05:22.178440 17268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:05:22.178512 17268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:22.178586 17268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:05:22.178637 17268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:22.188062 17268 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.67:37171
I20260810 18:05:22.188122 17400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.67:37171 every 8 connection(s)
I20260810 18:05:22.189132 17268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/data/info.pb
I20260810 18:05:22.193513 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 17268
I20260810 18:05:22.193647 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/wal/instance
I20260810 18:05:22.194000 29005 external_mini_cluster.cc:1428] Setting key 8c26817b3bb70567977408b99843cd6f
I20260810 18:05:22.199719 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.191343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:39551 (local address 127.28.83.126:44197)
0810 18:05:22.191543 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:22.191549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:22.193063 (+ 1514us) server_negotiation.cc:407] Connection header received
0810 18:05:22.194853 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:22.194858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:22.194914 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:22.195001 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:22.196003 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.196633 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.197838 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.198056 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.198142 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.199179 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:22.199182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:22.199185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:22.199198 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:22.199217 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:22.199225 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:22.199283 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:22.199449 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 18:05:22.199532 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 18:05:22.200675 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.191821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:44197 (local address 127.28.83.67:39551)
0810 18:05:22.192921 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:22.192964 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:22.194667 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:22.195188 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:22.195195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:22.195304 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:22.195869 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.195883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.196763 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.196766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.197723 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.197733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.198191 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.198193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.198342 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.198932 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:05:22.198934 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:22.198973 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:05:22.199039 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:22.199308 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:22.199318 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:22.199326 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:05:22.199432 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 18:05:22.199726 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":212,"threads_started":1}
I20260810 18:05:22.201889 17401 heartbeater.cc:347] Connected to a master server at 127.28.83.126:44197
I20260810 18:05:22.202119 17401 heartbeater.cc:464] Registering TS with master...
I20260810 18:05:22.202651 17401 heartbeater.cc:511] Master 127.28.83.126:44197 requested a full tablet report, sending...
I20260810 18:05:22.203639 16928 ts_manager.cc:194] Registered new tserver with Master: dc84a87d3ede4edea800dca92b091d8a (127.28.83.67:37171)
I20260810 18:05:22.204226 16928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.83.67:39551
I20260810 18:05:22.208060 29005 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:05:22.214097 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.210091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.28.83.126:44197)
0810 18:05:22.210252 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:22.210258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:22.210429 (+ 171us) server_negotiation.cc:407] Connection header received
0810 18:05:22.210601 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:22.210604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:22.210655 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:22.210739 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:22.211020 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.211531 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.212402 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.212621 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.212725 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.213684 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:22.213688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:22.213692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:22.213708 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:22.213732 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:22.213742 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:22.213814 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:22.213887 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 18:05:22.213945 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:05:22.217024 16928 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:05:22.221194 16928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:05:22.235824 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.230516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.67:37171 (local address 127.0.0.1:53090)
0810 18:05:22.231196 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:22.231208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:22.231282 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:22.231919 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:22.231922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:22.231938 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:22.232028 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.232034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.232966 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.232971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.234407 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.234418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.235275 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.235278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.235382 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.235419 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 18:05:22.235537 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 18:05:22.235665 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":18,"thread_start_us":85,"threads_started":1}
I20260810 18:05:22.235831 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.230739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.28.83.67:37171)
0810 18:05:22.231492 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:22.231524 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:22.231549 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:05:22.231627 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:22.231632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:22.231803 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:22.231947 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:22.232120 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.232854 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.234557 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.235138 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.235263 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.235462 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:22.235564 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 18:05:22.235661 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":150,"threads_started":1}
I20260810 18:05:22.236083 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.230100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41097 (local address 127.0.0.1:33424)
0810 18:05:22.230942 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:22.230955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:22.231030 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:22.232007 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:22.232010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:22.232029 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:22.232122 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.232127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.233706 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.233709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.234650 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.234661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.235674 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.235677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.235767 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.235800 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 18:05:22.235851 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 18:05:22.235917 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":137,"threads_started":1}
I20260810 18:05:22.236791 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.229961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.28.83.65:42269)
0810 18:05:22.230356 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:22.230367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:22.231084 (+ 717us) server_negotiation.cc:407] Connection header received
0810 18:05:22.231174 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:22.231179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:22.231417 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:22.231573 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:22.231884 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.233078 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.234646 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.235450 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.235559 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.235767 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:22.236510 (+ 743us) server_negotiation.cc:299] Negotiation successful
0810 18:05:22.236608 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":178,"threads_started":1}
I20260810 18:05:22.236851 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.230916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.28.83.66:41097)
0810 18:05:22.231545 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:22.231557 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:22.231579 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:05:22.231651 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:22.231656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:22.231843 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:22.232274 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:22.232406 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.233575 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.234801 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.235545 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.236361 (+ 816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.236550 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:22.236597 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 18:05:22.236694 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":99,"threads_started":1}
I20260810 18:05:22.236655 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.229793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42269 (local address 127.0.0.1:51424)
0810 18:05:22.230729 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:22.230797 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:22.230935 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:22.231586 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:22.231592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:22.231616 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:22.231755 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.231761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.233192 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.233196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.234522 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.234530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.235827 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.235834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.236360 (+ 526us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.236384 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:05:22.236462 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:05:22.236530 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":5,"thread_start_us":83,"threads_started":1}
I20260810 18:05:22.239151 17333 tablet_service.cc:1467] Processing CreateTablet for tablet 502188cd6760487d8651c37a543c6f37 (DEFAULT_TABLE table=test-table [id=112b57858d554ba5b73ebc9e3f8d6be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:22.239387 17196 tablet_service.cc:1467] Processing CreateTablet for tablet 502188cd6760487d8651c37a543c6f37 (DEFAULT_TABLE table=test-table [id=112b57858d554ba5b73ebc9e3f8d6be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:22.239500 17055 tablet_service.cc:1467] Processing CreateTablet for tablet 502188cd6760487d8651c37a543c6f37 (DEFAULT_TABLE table=test-table [id=112b57858d554ba5b73ebc9e3f8d6be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:05:22.240351 17333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502188cd6760487d8651c37a543c6f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:22.240351 17196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502188cd6760487d8651c37a543c6f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:22.240449 17055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502188cd6760487d8651c37a543c6f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:22.247344 17419 tablet_bootstrap.cc:492] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4: Bootstrap starting.
I20260810 18:05:22.249104 17420 tablet_bootstrap.cc:492] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: Bootstrap starting.
I20260810 18:05:22.249717 17422 tablet_bootstrap.cc:492] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: Bootstrap starting.
I20260810 18:05:22.249881 17419 tablet_bootstrap.cc:654] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:22.250818 17419 log.cc:824] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:22.251487 17420 tablet_bootstrap.cc:654] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:22.251907 17422 tablet_bootstrap.cc:654] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:22.252358 17420 log.cc:824] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:22.252691 17422 log.cc:824] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:22.253619 17419 tablet_bootstrap.cc:492] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4: No bootstrap required, opened a new log
I20260810 18:05:22.253840 17419 ts_tablet_manager.cc:1397] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:05:22.254473 17420 tablet_bootstrap.cc:492] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: No bootstrap required, opened a new log
I20260810 18:05:22.254652 17420 ts_tablet_manager.cc:1397] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:05:22.258379 17419 raft_consensus.cc:348] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } }
I20260810 18:05:22.258667 17419 raft_consensus.cc:374] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:22.258962 17419 raft_consensus.cc:729] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c634b5450c9b4b8d82311be9276de9b4, State: Initialized, Role: FOLLOWER
I20260810 18:05:22.258813 17420 raft_consensus.cc:348] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } }
I20260810 18:05:22.259033 17422 tablet_bootstrap.cc:492] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: No bootstrap required, opened a new log
I20260810 18:05:22.259052 17420 raft_consensus.cc:374] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:22.259095 17420 raft_consensus.cc:729] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc84a87d3ede4edea800dca92b091d8a, State: Initialized, Role: FOLLOWER
I20260810 18:05:22.259310 17422 ts_tablet_manager.cc:1397] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260810 18:05:22.259509 17419 consensus_queue.cc:260] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } }
I20260810 18:05:22.259509 17420 consensus_queue.cc:260] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } }
I20260810 18:05:22.260543 17419 ts_tablet_manager.cc:1428] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:05:22.260543 17420 ts_tablet_manager.cc:1428] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:05:22.260941 17401 heartbeater.cc:503] Master 127.28.83.126:44197 was elected leader, sending a full tablet report...
I20260810 18:05:22.260941 17126 heartbeater.cc:503] Master 127.28.83.126:44197 was elected leader, sending a full tablet report...
I20260810 18:05:22.264176 17422 raft_consensus.cc:348] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } }
I20260810 18:05:22.264412 17422 raft_consensus.cc:374] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:22.264472 17422 raft_consensus.cc:729] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4138bdb316c748dba586c619481e9fd2, State: Initialized, Role: FOLLOWER
I20260810 18:05:22.264894 17422 consensus_queue.cc:260] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } }
I20260810 18:05:22.265630 17264 heartbeater.cc:503] Master 127.28.83.126:44197 was elected leader, sending a full tablet report...
I20260810 18:05:22.265957 17422 ts_tablet_manager.cc:1428] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260810 18:05:22.273991 17127 tablet.cc:2369] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:05:22.400170 17426 raft_consensus.cc:482] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:05:22.400444 17426 raft_consensus.cc:504] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } }
I20260810 18:05:22.401749 17426 leader_election.cc:288] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c634b5450c9b4b8d82311be9276de9b4 (127.28.83.65:42269), 4138bdb316c748dba586c619481e9fd2 (127.28.83.66:41097)
I20260810 18:05:22.405568 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.402150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:44907 (local address 127.28.83.66:41097)
0810 18:05:22.402286 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:22.402291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:22.402306 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:05:22.402473 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:22.402476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:22.402524 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:22.402615 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:22.402913 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.403446 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.404561 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.404977 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.405111 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.405162 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:22.405353 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 18:05:22.405453 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:05:22.405570 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.402066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41097 (local address 127.28.83.67:44907)
0810 18:05:22.402200 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:22.402217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:22.402322 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:22.402631 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:22.402634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:22.402650 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:22.402783 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.402789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.403573 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.403577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.404422 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.404431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.405109 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.405113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.405216 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.405245 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:05:22.405334 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 18:05:22.405404 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":27}
I20260810 18:05:22.405815 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.402161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.83.67:45775 (local address 127.28.83.65:42269)
0810 18:05:22.402290 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:22.402294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:22.402709 (+ 415us) server_negotiation.cc:407] Connection header received
0810 18:05:22.402874 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:22.402877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:22.402932 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:22.403020 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:05:22.403302 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.403797 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.404851 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.405246 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.405375 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.405441 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:22.405634 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 18:05:22.405701 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:05:22.405831 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.402077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42269 (local address 127.28.83.67:45775)
0810 18:05:22.402598 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:22.402610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:22.402753 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:22.403031 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:22.403034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:22.403049 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:05:22.403183 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.403188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.403913 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.403917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.404703 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:22.404720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.405368 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.405371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:22.405508 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.405526 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:05:22.405616 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:05:22.405681 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":121,"thread_start_us":161,"threads_started":1}
I20260810 18:05:22.406385 17216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502188cd6760487d8651c37a543c6f37" candidate_uuid: "dc84a87d3ede4edea800dca92b091d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4138bdb316c748dba586c619481e9fd2" is_pre_election: true
I20260810 18:05:22.406463 17077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502188cd6760487d8651c37a543c6f37" candidate_uuid: "dc84a87d3ede4edea800dca92b091d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c634b5450c9b4b8d82311be9276de9b4" is_pre_election: true
I20260810 18:05:22.406745 17216 raft_consensus.cc:2454] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc84a87d3ede4edea800dca92b091d8a in term 0.
I20260810 18:05:22.406745 17077 raft_consensus.cc:2454] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc84a87d3ede4edea800dca92b091d8a in term 0.
I20260810 18:05:22.407284 17285 leader_election.cc:302] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4138bdb316c748dba586c619481e9fd2, c634b5450c9b4b8d82311be9276de9b4, dc84a87d3ede4edea800dca92b091d8a; no voters:
I20260810 18:05:22.407547 17426 raft_consensus.cc:2781] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:05:22.407635 17426 raft_consensus.cc:482] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:05:22.407678 17426 raft_consensus.cc:3037] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:22.408867 17426 raft_consensus.cc:504] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } }
I20260810 18:05:22.409289 17426 leader_election.cc:288] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [CANDIDATE]: Term 1 election: Requested vote from peers c634b5450c9b4b8d82311be9276de9b4 (127.28.83.65:42269), 4138bdb316c748dba586c619481e9fd2 (127.28.83.66:41097)
I20260810 18:05:22.409731 17077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502188cd6760487d8651c37a543c6f37" candidate_uuid: "dc84a87d3ede4edea800dca92b091d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c634b5450c9b4b8d82311be9276de9b4"
I20260810 18:05:22.409837 17216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502188cd6760487d8651c37a543c6f37" candidate_uuid: "dc84a87d3ede4edea800dca92b091d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4138bdb316c748dba586c619481e9fd2"
I20260810 18:05:22.409910 17077 raft_consensus.cc:3037] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:22.409983 17216 raft_consensus.cc:3037] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:22.411077 17216 raft_consensus.cc:2454] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc84a87d3ede4edea800dca92b091d8a in term 1.
I20260810 18:05:22.411083 17077 raft_consensus.cc:2454] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc84a87d3ede4edea800dca92b091d8a in term 1.
I20260810 18:05:22.411453 17286 leader_election.cc:302] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4138bdb316c748dba586c619481e9fd2, dc84a87d3ede4edea800dca92b091d8a; no voters:
I20260810 18:05:22.411646 17426 raft_consensus.cc:2781] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:22.411906 17426 raft_consensus.cc:686] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 1 LEADER]: Becoming Leader. State: Replica: dc84a87d3ede4edea800dca92b091d8a, State: Running, Role: LEADER
I20260810 18:05:22.412276 17426 consensus_queue.cc:237] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } }
I20260810 18:05:22.415544 16927 catalog_manager.cc:5495] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to dc84a87d3ede4edea800dca92b091d8a (127.28.83.67). New cstate: current_term: 1 leader_uuid: "dc84a87d3ede4edea800dca92b091d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc84a87d3ede4edea800dca92b091d8a" member_type: VOTER last_known_addr { host: "127.28.83.67" port: 37171 } health_report { overall_health: HEALTHY } } }
I20260810 18:05:22.427446 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.423548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.28.83.67:37171)
0810 18:05:22.423711 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:22.423715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:22.423731 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:05:22.423894 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:22.423897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:22.423952 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:22.424082 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:22.424334 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.424895 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.425870 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:22.426099 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:22.426212 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:22.426532 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:22.426982 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:22.427121 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:22.427211 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 18:05:22.427267 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:05:22.433552 17077 raft_consensus.cc:1261] T 502188cd6760487d8651c37a543c6f37 P c634b5450c9b4b8d82311be9276de9b4 [term 1 FOLLOWER]: Refusing update from remote peer dc84a87d3ede4edea800dca92b091d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:22.433554 17216 raft_consensus.cc:1261] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [term 1 FOLLOWER]: Refusing update from remote peer dc84a87d3ede4edea800dca92b091d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:22.434387 17430 consensus_queue.cc:1035] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c634b5450c9b4b8d82311be9276de9b4" member_type: VOTER last_known_addr { host: "127.28.83.65" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:22.434674 17426 consensus_queue.cc:1035] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4138bdb316c748dba586c619481e9fd2" member_type: VOTER last_known_addr { host: "127.28.83.66" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:05:22.440227 17403 tablet.cc:2369] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:05:22.444646 17435 mvcc.cc:205] Tried to move back new op lower bound from 7317033461480038400 to 7317033461403512832. Current Snapshot: MvccSnapshot[applied={T|T < 7317033461480038400}]
I20260810 18:05:22.453864 16928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:22.454071 16928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:05:22.456724 16928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d44e25cce8b46aaa8412a9150f5e964: Sending DeleteTablet for 3 replicas of tablet 502188cd6760487d8651c37a543c6f37
I20260810 18:05:22.457764 17196 tablet_service.cc:1514] Processing DeleteTablet for tablet 502188cd6760487d8651c37a543c6f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:22 UTC) from {username='slave'} at 127.0.0.1:33424
I20260810 18:05:22.457764 17333 tablet_service.cc:1514] Processing DeleteTablet for tablet 502188cd6760487d8651c37a543c6f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:22 UTC) from {username='slave'} at 127.0.0.1:53090
I20260810 18:05:22.458540 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 16992
I20260810 18:05:22.458566 17055 tablet_service.cc:1514] Processing DeleteTablet for tablet 502188cd6760487d8651c37a543c6f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:05:22 UTC) from {username='slave'} at 127.0.0.1:51424
I20260810 18:05:22.458674 17444 tablet_replica.cc:330] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: stopping tablet replica
I20260810 18:05:22.459028 17444 raft_consensus.cc:2229] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:05:22.459404 17444 raft_consensus.cc:2258] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:22.460810 17444 ts_tablet_manager.cc:1905] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:05:22.463461 17444 ts_tablet_manager.cc:1918] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:22.463587 17444 log.cc:1196] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-2/wal/wals/502188cd6760487d8651c37a543c6f37
I20260810 18:05:22.463878 17444 ts_tablet_manager.cc:1939] T 502188cd6760487d8651c37a543c6f37 P dc84a87d3ede4edea800dca92b091d8a: Deleting consensus metadata
I20260810 18:05:22.465134 17443 tablet_replica.cc:330] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: stopping tablet replica
I20260810 18:05:22.465090 16913 catalog_manager.cc:4841] TS dc84a87d3ede4edea800dca92b091d8a (127.28.83.67:37171): tablet 502188cd6760487d8651c37a543c6f37 (table test-table [id=112b57858d554ba5b73ebc9e3f8d6be0]) successfully deleted
I20260810 18:05:22.465547 17443 raft_consensus.cc:2229] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:05:22.465886 17443 raft_consensus.cc:2258] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:05:22.467258 17443 ts_tablet_manager.cc:1905] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 18:05:22.468803 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.83.65:42269 (error 108)
I20260810 18:05:22.469920 17443 ts_tablet_manager.cc:1918] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:05:22.469959 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.469602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.65:42269 (local address 127.0.0.1:51434)
0810 18:05:22.469778 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:22.469860 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.65:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 18:05:22.470027 17443 log.cc:1196] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: Deleting WAL directory at /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786384863806381-29005-0/minicluster-data/ts-1/wal/wals/502188cd6760487d8651c37a543c6f37
I20260810 18:05:22.470013 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 17129
I20260810 18:05:22.470356 17443 ts_tablet_manager.cc:1939] T 502188cd6760487d8651c37a543c6f37 P 4138bdb316c748dba586c619481e9fd2: Deleting consensus metadata
W20260810 18:05:22.473078 16913 catalog_manager.cc:4584] TS c634b5450c9b4b8d82311be9276de9b4 (127.28.83.65:42269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 502188cd6760487d8651c37a543c6f37: Network error: Client connection negotiation failed: client connection to 127.28.83.65:42269: connect: Connection refused (error 111)
I20260810 18:05:22.479244 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 17268
I20260810 18:05:22.482071 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:22.481725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.66:41097 (local address 127.0.0.1:33428)
0810 18:05:22.481919 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:22.481982 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.66:41097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 18:05:22.486119 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 16897
2026-08-10T18:05:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1177 ms)
[----------] 6 tests from EncryptionPolicyTest (10701 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:22 dist-test-slave-srgv krb5kdc[17448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:22 dist-test-slave-srgv krb5kdc[17448](info): set up 2 sockets
Aug 10 18:05:22 dist-test-slave-srgv krb5kdc[17448](info): commencing operation
krb5kdc: starting...
W20260810 18:05:23.540578 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:23 dist-test-slave-srgv krb5kdc[17448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:23.670662 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:23.785316 17462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:23.785722 17462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:23.785768 17462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:23.789835 17462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:23.789903 17462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:23.789923 17462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:23.789940 17462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:23.789955 17462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:23.795598 17462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:23.796692 17462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:23.797708 17462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:23.806159 17472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:23.806224 17470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:23.806397 17462 server_base.cc:1034] running on GCE node
W20260810 18:05:23.806170 17469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:23.807101 17462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:23.808300 17462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:23.809486 17462 hybrid_clock.cc:648] HybridClock initialized: now 1786385123809471 us; error 29 us; skew 500 ppm
Aug 10 18:05:23 dist-test-slave-srgv krb5kdc[17448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:23.812390 17462 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:23.813359 17462 webserver.cc:460] Webserver started at http://127.0.0.1:34867/ using document root <none> and password file <none>
I20260810 18:05:23.814002 17462 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:23.814098 17462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:23.814352 17462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:23.816558 17462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "d03315fb7351439bb3f9eabb729af35b"
format_stamp: "Formatted at 2026-08-10 18:05:23 on dist-test-slave-srgv"
server_key: "edd698416ddeca374f9753b15f932e5a"
server_key_iv: "6eae1e42547a6475e3359a76fd314e9d"
server_key_version: "encryptionkey@0"
I20260810 18:05:23.817180 17462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "d03315fb7351439bb3f9eabb729af35b"
format_stamp: "Formatted at 2026-08-10 18:05:23 on dist-test-slave-srgv"
server_key: "edd698416ddeca374f9753b15f932e5a"
server_key_iv: "6eae1e42547a6475e3359a76fd314e9d"
server_key_version: "encryptionkey@0"
I20260810 18:05:23.821108 17462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:05:23.823817 17479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:23.825112 17462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:05:23.825219 17462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "d03315fb7351439bb3f9eabb729af35b"
format_stamp: "Formatted at 2026-08-10 18:05:23 on dist-test-slave-srgv"
server_key: "edd698416ddeca374f9753b15f932e5a"
server_key_iv: "6eae1e42547a6475e3359a76fd314e9d"
server_key_version: "encryptionkey@0"
I20260810 18:05:23.825314 17462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:23.859573 17462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:23.862931 17462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:23.863121 17462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:23.870508 17462 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36323
I20260810 18:05:23.870522 17531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36323 every 8 connection(s)
I20260810 18:05:23.871543 17462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:23.874562 17533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:23.878535 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 17462
I20260810 18:05:23.878649 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:23.878950 29005 external_mini_cluster.cc:1428] Setting key c7fcb26b47f4e01d65bd799b75b90470
I20260810 18:05:23.880725 17533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b: Bootstrap starting.
I20260810 18:05:23.883383 17533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:23.884191 17533 log.cc:824] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:23.886549 17533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b: No bootstrap required, opened a new log
Aug 10 18:05:23 dist-test-slave-srgv krb5kdc[17448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:05:23.890302 17533 raft_consensus.cc:348] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03315fb7351439bb3f9eabb729af35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260810 18:05:23.890622 17533 raft_consensus.cc:374] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:23.890722 17533 raft_consensus.cc:729] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03315fb7351439bb3f9eabb729af35b, State: Initialized, Role: FOLLOWER
I20260810 18:05:23.891261 17533 consensus_queue.cc:260] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03315fb7351439bb3f9eabb729af35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260810 18:05:23.891434 17533 raft_consensus.cc:388] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:23.891503 17533 raft_consensus.cc:482] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:23.891609 17533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:23.892925 17533 raft_consensus.cc:504] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03315fb7351439bb3f9eabb729af35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260810 18:05:23.893321 17533 leader_election.cc:302] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d03315fb7351439bb3f9eabb729af35b; no voters:
I20260810 18:05:23.893747 17533 leader_election.cc:288] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:23.893941 17538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:23.894188 17538 raft_consensus.cc:686] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [term 1 LEADER]: Becoming Leader. State: Replica: d03315fb7351439bb3f9eabb729af35b, State: Running, Role: LEADER
I20260810 18:05:23.894541 17538 consensus_queue.cc:237] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03315fb7351439bb3f9eabb729af35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260810 18:05:23.894971 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:23.880392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43529 (local address 127.0.0.1:36323)
0810 18:05:23.880817 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:23.880834 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:23.880902 (+ 68us) server_negotiation.cc:407] Connection header received
0810 18:05:23.881641 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:23.881659 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:23.882004 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:23.882286 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:23.883252 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:23.884363 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:23.885511 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:23.885980 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:23.886100 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:23.889054 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:23.889060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:23.889073 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:23.889100 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:23.891519 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:23.892174 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:23.892179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:23.892183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:23.892345 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:23.892708 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:23.892710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:23.892712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:23.893160 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:23.893326 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:23.893692 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 18:05:23.893959 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":138,"threads_started":1}
I20260810 18:05:23.895066 17533 sys_catalog.cc:564] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:23.896168 17539 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d03315fb7351439bb3f9eabb729af35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03315fb7351439bb3f9eabb729af35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } } }
I20260810 18:05:23.896296 17539 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:23.896520 17539 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03315fb7351439bb3f9eabb729af35b. Latest consensus state: current_term: 1 leader_uuid: "d03315fb7351439bb3f9eabb729af35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03315fb7351439bb3f9eabb729af35b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } } }
I20260810 18:05:23.896595 17539 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:23.897286 17548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:23.898375 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:23.901034 17548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:23.907590 17548 catalog_manager.cc:1347] Generated new cluster ID: 5ec0228b052545fbaf402a68a22ede6b
I20260810 18:05:23.907667 17548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:23.909447 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:23.900462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41305 (local address 127.0.0.1:36323)
0810 18:05:23.900580 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:23.900584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:23.900772 (+ 188us) server_negotiation.cc:407] Connection header received
0810 18:05:23.900915 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:23.900918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:23.900962 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:23.901057 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:23.901818 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:23.902374 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:23.903300 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:23.903669 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:23.903780 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:23.905616 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:23.905620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:23.905623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:23.905655 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:23.907413 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:23.908014 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:23.908018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:23.908020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:23.908120 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:23.908464 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:23.908467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:23.908469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:23.908684 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:23.908842 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:23.909136 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 18:05:23.909236 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 18:05:23.948494 17548 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:23.949887 17548 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:23.954782 17548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d03315fb7351439bb3f9eabb729af35b: Generated new TSK 0
I20260810 18:05:23.955511 17548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:23.962352 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 17462
W20260810 18:05:23.969488 17446 connection.cc:535] client connection to 127.0.0.1:36323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36323): Connection reset by peer (error 104)
2026-08-10T18:05:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:24 dist-test-slave-srgv krb5kdc[17566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:24 dist-test-slave-srgv krb5kdc[17566](info): set up 2 sockets
Aug 10 18:05:24 dist-test-slave-srgv krb5kdc[17566](info): commencing operation
krb5kdc: starting...
W20260810 18:05:25.028309 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:25 dist-test-slave-srgv krb5kdc[17566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:25.165567 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:25.284498 17579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:25.284854 17579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:25.284937 17579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:25.289166 17579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:25.289302 17579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:25.289341 17579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:25.289382 17579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:25.289463 17579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:25.295217 17579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:25.296586 17579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:25.297695 17579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:25.306514 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:25.306713 17584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:25.306517 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:25.306887 17579 server_base.cc:1034] running on GCE node
I20260810 18:05:25.307565 17579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:25.308755 17579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:25.309978 17579 hybrid_clock.cc:648] HybridClock initialized: now 1786385125309933 us; error 68 us; skew 500 ppm
Aug 10 18:05:25 dist-test-slave-srgv krb5kdc[17566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:25.319640 17579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:25.320629 17579 webserver.cc:460] Webserver started at http://127.0.0.1:33589/ using document root <none> and password file <none>
I20260810 18:05:25.321187 17579 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:25.321251 17579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:25.321498 17579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:25.323668 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "e3b9a56514d44d73ac144e052d15856c"
format_stamp: "Formatted at 2026-08-10 18:05:25 on dist-test-slave-srgv"
server_key: "415010411413a347c972ab00176c2cc9"
server_key_iv: "36420276fffa79a74b93e97bb7495b14"
server_key_version: "encryptionkey@0"
I20260810 18:05:25.324230 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "e3b9a56514d44d73ac144e052d15856c"
format_stamp: "Formatted at 2026-08-10 18:05:25 on dist-test-slave-srgv"
server_key: "415010411413a347c972ab00176c2cc9"
server_key_iv: "36420276fffa79a74b93e97bb7495b14"
server_key_version: "encryptionkey@0"
I20260810 18:05:25.328038 17579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 18:05:25.330578 17594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:25.331890 17579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:05:25.332043 17579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "e3b9a56514d44d73ac144e052d15856c"
format_stamp: "Formatted at 2026-08-10 18:05:25 on dist-test-slave-srgv"
server_key: "415010411413a347c972ab00176c2cc9"
server_key_iv: "36420276fffa79a74b93e97bb7495b14"
server_key_version: "encryptionkey@0"
I20260810 18:05:25.332194 17579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:25.369155 17579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:25.372797 17579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:25.373027 17579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:25.380563 17579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44353
I20260810 18:05:25.380599 17650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44353 every 8 connection(s)
I20260810 18:05:25.381574 17579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:25.383883 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 17579
I20260810 18:05:25.384052 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:25.384403 29005 external_mini_cluster.cc:1428] Setting key 6b7a3a6b3e39896de358812a3d4606e3
I20260810 18:05:25.384719 17651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:25.391654 17651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c: Bootstrap starting.
I20260810 18:05:25.394160 17654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:25.386020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49451 (local address 127.0.0.1:44353)
0810 18:05:25.386487 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:25.386504 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:25.386540 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:05:25.387613 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:25.387632 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:25.387982 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:25.388323 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:25.388741 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:25.389810 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:25.390727 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:25.391213 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:25.391388 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:25.392387 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:25.392391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:25.392404 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:25.392799 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:25.392856 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:25.392878 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:25.393023 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:25.393218 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 18:05:25.393364 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":153,"threads_started":1}
I20260810 18:05:25.394378 17651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:25.395162 17651 log.cc:824] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:25.397297 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:25.397658 17651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c: No bootstrap required, opened a new log
I20260810 18:05:25.401611 17651 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b9a56514d44d73ac144e052d15856c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260810 18:05:25.401916 17651 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:25.401999 17651 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b9a56514d44d73ac144e052d15856c, State: Initialized, Role: FOLLOWER
I20260810 18:05:25.402503 17651 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b9a56514d44d73ac144e052d15856c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260810 18:05:25.402663 17651 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:25.402757 17651 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:25.402873 17651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:25.403764 17654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:25.399511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51443 (local address 127.0.0.1:44353)
0810 18:05:25.399665 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:25.399676 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:25.399888 (+ 212us) server_negotiation.cc:407] Connection header received
0810 18:05:25.400033 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:25.400036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:25.400089 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:25.400172 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:25.400439 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:25.400953 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:25.401842 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:25.402230 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:25.402338 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:25.403263 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:25.403267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:25.403269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:25.403282 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:25.403314 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:25.403325 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:25.403422 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:25.403572 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 18:05:25.403606 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 18:05:25.403935 17651 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b9a56514d44d73ac144e052d15856c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260810 18:05:25.404364 17651 leader_election.cc:302] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3b9a56514d44d73ac144e052d15856c; no voters:
I20260810 18:05:25.404732 17651 leader_election.cc:288] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:25.404870 17661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:25.405118 17661 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [term 1 LEADER]: Becoming Leader. State: Replica: e3b9a56514d44d73ac144e052d15856c, State: Running, Role: LEADER
I20260810 18:05:25.405480 17661 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b9a56514d44d73ac144e052d15856c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260810 18:05:25.406092 17651 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:25.407258 17662 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3b9a56514d44d73ac144e052d15856c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b9a56514d44d73ac144e052d15856c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } } }
I20260810 18:05:25.407269 17663 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3b9a56514d44d73ac144e052d15856c. Latest consensus state: current_term: 1 leader_uuid: "e3b9a56514d44d73ac144e052d15856c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b9a56514d44d73ac144e052d15856c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } } }
I20260810 18:05:25.407407 17663 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:25.407407 17662 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:25.407984 17670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:25.411206 17670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:25.419778 17670 catalog_manager.cc:1347] Generated new cluster ID: 97b9047485044f7296f9e0b5fad9a17a
I20260810 18:05:25.419868 17670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:25.456993 17670 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:25.458487 17670 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:25.466485 17670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3b9a56514d44d73ac144e052d15856c: Generated new TSK 0
I20260810 18:05:25.467376 17670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:25.474296 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 17579
W20260810 18:05:25.480851 17564 connection.cc:535] client connection to 127.0.0.1:44353 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44353): Connection reset by peer (error 104)
2026-08-10T18:05:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1511 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:25 dist-test-slave-srgv krb5kdc[17683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:25 dist-test-slave-srgv krb5kdc[17683](info): set up 2 sockets
Aug 10 18:05:25 dist-test-slave-srgv krb5kdc[17683](info): commencing operation
krb5kdc: starting...
W20260810 18:05:26.532405 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:26 dist-test-slave-srgv krb5kdc[17683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:26.662824 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:26.775665 17696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:26.775945 17696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:26.775987 17696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:26.780099 17696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:26.780172 17696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:26.780191 17696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:26.780210 17696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:26.780224 17696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:26.785734 17696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:26.786916 17696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:26.787987 17696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:26.797144 17701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:26.797334 17696 server_base.cc:1034] running on GCE node
W20260810 18:05:26.797154 17702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:26.797629 17705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:26.798317 17696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:26.799517 17696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:26.800726 17696 hybrid_clock.cc:648] HybridClock initialized: now 1786385126800702 us; error 55 us; skew 500 ppm
Aug 10 18:05:26 dist-test-slave-srgv krb5kdc[17683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:26.803689 17696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:26.804603 17696 webserver.cc:460] Webserver started at http://127.0.0.1:34869/ using document root <none> and password file <none>
I20260810 18:05:26.805140 17696 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:26.805202 17696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:26.805368 17696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:26.807468 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "ffff7a7540e34b54a21fdceb68dd927c"
format_stamp: "Formatted at 2026-08-10 18:05:26 on dist-test-slave-srgv"
server_key: "365bcabac657ef6615e47c41f9d0377d"
server_key_iv: "875f0dfea0634766c12bafc53355cff5"
server_key_version: "encryptionkey@0"
I20260810 18:05:26.807984 17696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "ffff7a7540e34b54a21fdceb68dd927c"
format_stamp: "Formatted at 2026-08-10 18:05:26 on dist-test-slave-srgv"
server_key: "365bcabac657ef6615e47c41f9d0377d"
server_key_iv: "875f0dfea0634766c12bafc53355cff5"
server_key_version: "encryptionkey@0"
I20260810 18:05:26.811769 17696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:26.814239 17714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:26.815568 17696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:05:26.815701 17696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "ffff7a7540e34b54a21fdceb68dd927c"
format_stamp: "Formatted at 2026-08-10 18:05:26 on dist-test-slave-srgv"
server_key: "365bcabac657ef6615e47c41f9d0377d"
server_key_iv: "875f0dfea0634766c12bafc53355cff5"
server_key_version: "encryptionkey@0"
I20260810 18:05:26.815809 17696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:26.820508 17696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:26.823865 17696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:26.824085 17696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:26.831295 17696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46627
I20260810 18:05:26.831362 17768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46627 every 8 connection(s)
I20260810 18:05:26.832324 17696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:26.835299 17769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:26.840555 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 17696
I20260810 18:05:26.840711 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:26.840984 29005 external_mini_cluster.cc:1428] Setting key 1c71e090ec7dc54c3fce566bd3fa1d57
I20260810 18:05:26.841212 17769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c: Bootstrap starting.
I20260810 18:05:26.844528 17769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:26.845324 17769 log.cc:824] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:26.846710 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:26.842472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44991 (local address 127.0.0.1:46627)
0810 18:05:26.843140 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:26.843154 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:26.843190 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:05:26.843839 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:26.843855 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:26.844156 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:26.844486 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:26.845523 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:26.845529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:26.845544 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:26.845554 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:26.845609 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:26.845634 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:26.845754 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:26.845858 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 18:05:26.846011 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":178,"threads_started":1}
I20260810 18:05:26.847427 17769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c: No bootstrap required, opened a new log
I20260810 18:05:26.849443 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:26.851524 17769 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffff7a7540e34b54a21fdceb68dd927c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } }
I20260810 18:05:26.851846 17769 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:26.851955 17769 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffff7a7540e34b54a21fdceb68dd927c, State: Initialized, Role: FOLLOWER
I20260810 18:05:26.852420 17769 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffff7a7540e34b54a21fdceb68dd927c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } }
I20260810 18:05:26.852581 17769 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:26.852684 17769 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:26.852797 17769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:26.853312 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:26.851559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49571 (local address 127.0.0.1:46627)
0810 18:05:26.851681 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:26.851685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:26.851864 (+ 179us) server_negotiation.cc:407] Connection header received
0810 18:05:26.851938 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:26.851942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:26.851983 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:26.852076 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:26.852891 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:26.852896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:26.852901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:26.852915 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:26.852944 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:26.852956 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:26.853035 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:26.853160 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 18:05:26.853191 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:05:26.854144 17769 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffff7a7540e34b54a21fdceb68dd927c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } }
I20260810 18:05:26.854491 17769 leader_election.cc:302] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffff7a7540e34b54a21fdceb68dd927c; no voters:
I20260810 18:05:26.854807 17769 leader_election.cc:288] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:26.854920 17780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:26.855159 17780 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [term 1 LEADER]: Becoming Leader. State: Replica: ffff7a7540e34b54a21fdceb68dd927c, State: Running, Role: LEADER
I20260810 18:05:26.855520 17780 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffff7a7540e34b54a21fdceb68dd927c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } }
I20260810 18:05:26.856115 17769 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:26.857206 17783 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffff7a7540e34b54a21fdceb68dd927c. Latest consensus state: current_term: 1 leader_uuid: "ffff7a7540e34b54a21fdceb68dd927c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffff7a7540e34b54a21fdceb68dd927c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } } }
I20260810 18:05:26.857336 17783 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:26.857511 17781 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffff7a7540e34b54a21fdceb68dd927c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffff7a7540e34b54a21fdceb68dd927c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46627 } } }
I20260810 18:05:26.857607 17781 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c [sys.catalog]: This master's current role is: LEADER
W20260810 18:05:26.861032 17796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:26.861137 17796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:05:26.861212 17797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:26.861997 17797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:26.867878 17797 catalog_manager.cc:1347] Generated new cluster ID: 3f43844f87b640d38e3dd26686131ab2
I20260810 18:05:26.867967 17797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:26.875501 17797 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:26.875921 17797 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:26.883018 17797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffff7a7540e34b54a21fdceb68dd927c: Generated new TSK 0
I20260810 18:05:26.883749 17797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:26.889968 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 17696
2026-08-10T18:05:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:26 dist-test-slave-srgv krb5kdc[17804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:26 dist-test-slave-srgv krb5kdc[17804](info): set up 2 sockets
Aug 10 18:05:26 dist-test-slave-srgv krb5kdc[17804](info): commencing operation
krb5kdc: starting...
W20260810 18:05:27.951695 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:28 dist-test-slave-srgv krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:28.094906 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:28.210529 17818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:28.210850 17818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:28.210893 17818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:28.215425 17818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:28.215593 17818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:28.215699 17818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:28.215744 17818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:28.215773 17818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:28.222461 17818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:28.223665 17818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:28.224763 17818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:28.234191 17826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:28.234226 17824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:28.234226 17823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:28.234500 17818 server_base.cc:1034] running on GCE node
I20260810 18:05:28.235116 17818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:28.236304 17818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:28.237489 17818 hybrid_clock.cc:648] HybridClock initialized: now 1786385128237464 us; error 48 us; skew 500 ppm
Aug 10 18:05:28 dist-test-slave-srgv krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:28.240504 17818 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:28.241616 17818 webserver.cc:460] Webserver started at http://127.0.0.1:38949/ using document root <none> and password file <none>
I20260810 18:05:28.242215 17818 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:28.242322 17818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:28.242584 17818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:28.244755 17818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "b1645a7bfcb84b89b80a3a06c1020311"
format_stamp: "Formatted at 2026-08-10 18:05:28 on dist-test-slave-srgv"
server_key: "60395c70cf99ac0db9cdb06382f9a0d1"
server_key_iv: "0a3cc1f49f6a543d5fe13f25d269b248"
server_key_version: "encryptionkey@0"
I20260810 18:05:28.245358 17818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "b1645a7bfcb84b89b80a3a06c1020311"
format_stamp: "Formatted at 2026-08-10 18:05:28 on dist-test-slave-srgv"
server_key: "60395c70cf99ac0db9cdb06382f9a0d1"
server_key_iv: "0a3cc1f49f6a543d5fe13f25d269b248"
server_key_version: "encryptionkey@0"
I20260810 18:05:28.249306 17818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 18:05:28.251869 17833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:28.253139 17818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:05:28.253305 17818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "b1645a7bfcb84b89b80a3a06c1020311"
format_stamp: "Formatted at 2026-08-10 18:05:28 on dist-test-slave-srgv"
server_key: "60395c70cf99ac0db9cdb06382f9a0d1"
server_key_iv: "0a3cc1f49f6a543d5fe13f25d269b248"
server_key_version: "encryptionkey@0"
I20260810 18:05:28.253479 17818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:28.276747 17818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:28.280104 17818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:28.280339 17818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:28.288060 17818 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43967
I20260810 18:05:28.288086 17892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43967 every 8 connection(s)
I20260810 18:05:28.289088 17818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:28.291880 17893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:28.292531 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 17818
I20260810 18:05:28.292682 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:28.292990 29005 external_mini_cluster.cc:1428] Setting key 4a13765ae5b3862793e79a49a8d38afb
I20260810 18:05:28.298053 17893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311: Bootstrap starting.
I20260810 18:05:28.300274 17893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:28.301015 17893 log.cc:824] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:05:28 dist-test-slave-srgv krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:05:28.303241 17893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311: No bootstrap required, opened a new log
I20260810 18:05:28.306990 17893 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1645a7bfcb84b89b80a3a06c1020311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43967 } }
I20260810 18:05:28.307217 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:28.294495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37847 (local address 127.0.0.1:43967)
0810 18:05:28.294965 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:28.294980 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:28.295016 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:05:28.295648 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:28.295665 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:28.295990 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:28.296338 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:28.297162 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:28.298228 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:28.299118 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:28.299596 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:28.299771 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:28.302411 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:28.302416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:28.302429 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:28.302456 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:28.304300 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:28.304756 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:28.304771 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:28.304776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:28.304887 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:28.305180 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:28.305183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:28.305185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:28.305659 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:28.305862 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:28.306153 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 18:05:28.306459 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":145,"threads_started":1}
I20260810 18:05:28.307217 17893 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:28.307387 17893 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1645a7bfcb84b89b80a3a06c1020311, State: Initialized, Role: FOLLOWER
I20260810 18:05:28.307817 17893 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1645a7bfcb84b89b80a3a06c1020311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43967 } }
I20260810 18:05:28.307945 17893 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:28.308030 17893 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:28.308132 17893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:28.309376 17893 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1645a7bfcb84b89b80a3a06c1020311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43967 } }
I20260810 18:05:28.309814 17893 leader_election.cc:302] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1645a7bfcb84b89b80a3a06c1020311; no voters:
I20260810 18:05:28.310185 17893 leader_election.cc:288] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:28.310348 17899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:28.310375 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:28.310616 17899 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [term 1 LEADER]: Becoming Leader. State: Replica: b1645a7bfcb84b89b80a3a06c1020311, State: Running, Role: LEADER
I20260810 18:05:28.310941 17899 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1645a7bfcb84b89b80a3a06c1020311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43967 } }
I20260810 18:05:28.311612 17893 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:28.312146 17899 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1645a7bfcb84b89b80a3a06c1020311. Latest consensus state: current_term: 1 leader_uuid: "b1645a7bfcb84b89b80a3a06c1020311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1645a7bfcb84b89b80a3a06c1020311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43967 } } }
I20260810 18:05:28.312283 17899 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:28.312592 17901 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1645a7bfcb84b89b80a3a06c1020311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1645a7bfcb84b89b80a3a06c1020311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43967 } } }
I20260810 18:05:28.312670 17901 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:28.312697 17909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:28.315939 17909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:28.322901 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:28.313513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60213 (local address 127.0.0.1:43967)
0810 18:05:28.313630 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:28.313636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:28.313990 (+ 354us) server_negotiation.cc:407] Connection header received
0810 18:05:28.314136 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:28.314155 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:28.314211 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:28.314296 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:28.315175 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:28.315900 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:28.317072 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:28.317428 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:28.317538 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:28.319370 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:28.319373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:28.319375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:28.319399 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:28.320917 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:28.321504 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:28.321507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:28.321510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:28.321605 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:28.321940 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:28.321943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:28.321945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:28.322166 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:28.322309 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:28.322584 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 18:05:28.322716 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 18:05:28.326275 17909 catalog_manager.cc:1347] Generated new cluster ID: f96c33abbecc47d2b4f9d5944d55c30d
I20260810 18:05:28.326366 17909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:28.339386 17909 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:28.340574 17909 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:28.348122 17909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1645a7bfcb84b89b80a3a06c1020311: Generated new TSK 0
I20260810 18:05:28.348933 17909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:28.360512 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 17818
2026-08-10T18:05:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:28 dist-test-slave-srgv krb5kdc[17926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:28 dist-test-slave-srgv krb5kdc[17926](info): set up 2 sockets
Aug 10 18:05:28 dist-test-slave-srgv krb5kdc[17926](info): commencing operation
krb5kdc: starting...
W20260810 18:05:29.416761 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:29 dist-test-slave-srgv krb5kdc[17926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:29.545173 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:29.660169 17940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:29.660532 17940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:29.660624 17940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:29.664772 17940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:29.664888 17940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:29.664928 17940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:29.664990 17940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:29.665030 17940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:29.670545 17940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:29.671753 17940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:29.672847 17940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:29.681613 17950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:29.681566 17947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:29.681567 17946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:29.682013 17940 server_base.cc:1034] running on GCE node
I20260810 18:05:29.682659 17940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:29.683825 17940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:29.685024 17940 hybrid_clock.cc:648] HybridClock initialized: now 1786385129685019 us; error 46 us; skew 500 ppm
Aug 10 18:05:29 dist-test-slave-srgv krb5kdc[17926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:29.687975 17940 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:29.688967 17940 webserver.cc:460] Webserver started at http://127.0.0.1:45929/ using document root <none> and password file <none>
I20260810 18:05:29.689604 17940 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:29.689699 17940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:29.689939 17940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:29.692013 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "294f1b2db5be40ebada87c8d21692036"
format_stamp: "Formatted at 2026-08-10 18:05:29 on dist-test-slave-srgv"
server_key: "d6e92fe9f0657639ec3200066b819003"
server_key_iv: "88959dd2c4e37b6de8770ea3b13a93df"
server_key_version: "encryptionkey@0"
I20260810 18:05:29.692617 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "294f1b2db5be40ebada87c8d21692036"
format_stamp: "Formatted at 2026-08-10 18:05:29 on dist-test-slave-srgv"
server_key: "d6e92fe9f0657639ec3200066b819003"
server_key_iv: "88959dd2c4e37b6de8770ea3b13a93df"
server_key_version: "encryptionkey@0"
I20260810 18:05:29.696403 17940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 18:05:29.698889 17957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:29.700251 17940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:05:29.700412 17940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "294f1b2db5be40ebada87c8d21692036"
format_stamp: "Formatted at 2026-08-10 18:05:29 on dist-test-slave-srgv"
server_key: "d6e92fe9f0657639ec3200066b819003"
server_key_iv: "88959dd2c4e37b6de8770ea3b13a93df"
server_key_version: "encryptionkey@0"
I20260810 18:05:29.700541 17940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:29.712338 17940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:29.715251 17940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:29.715461 17940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:29.723006 17940 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35969
I20260810 18:05:29.723032 18015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35969 every 8 connection(s)
I20260810 18:05:29.723990 17940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:29.726868 18016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:29.732331 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 17940
I20260810 18:05:29.732506 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:29.732622 18016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036: Bootstrap starting.
I20260810 18:05:29.732826 29005 external_mini_cluster.cc:1428] Setting key fcc305c3da4f5c13c6182a2c41abba29
I20260810 18:05:29.735301 18016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:29.736084 18016 log.cc:824] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:29.738029 18016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036: No bootstrap required, opened a new log
I20260810 18:05:29.741657 18016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294f1b2db5be40ebada87c8d21692036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } }
I20260810 18:05:29.741896 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:29.734260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58533 (local address 127.0.0.1:35969)
0810 18:05:29.734714 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:29.734729 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:29.734763 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:05:29.735470 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:29.735491 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:29.735835 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:29.736170 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:29.736590 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:29.737627 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:29.738612 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:29.739090 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:29.739282 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:29.740139 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:29.740144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:29.740158 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:29.740550 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:29.740606 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:29.740637 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:29.740747 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:29.740864 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 18:05:29.741082 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":141,"threads_started":1}
I20260810 18:05:29.741909 18016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:29.742158 18016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294f1b2db5be40ebada87c8d21692036, State: Initialized, Role: FOLLOWER
I20260810 18:05:29.742650 18016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294f1b2db5be40ebada87c8d21692036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } }
I20260810 18:05:29.742795 18016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:29.742847 18016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:29.742934 18016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:29.743968 18016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294f1b2db5be40ebada87c8d21692036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } }
I20260810 18:05:29.744334 18016 leader_election.cc:302] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 294f1b2db5be40ebada87c8d21692036; no voters:
I20260810 18:05:29.744663 18016 leader_election.cc:288] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:29.744797 18021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:29.745072 18021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [term 1 LEADER]: Becoming Leader. State: Replica: 294f1b2db5be40ebada87c8d21692036, State: Running, Role: LEADER
I20260810 18:05:29.745426 18021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294f1b2db5be40ebada87c8d21692036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } }
I20260810 18:05:29.745726 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:29.746133 18016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:29.747167 18023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 294f1b2db5be40ebada87c8d21692036. Latest consensus state: current_term: 1 leader_uuid: "294f1b2db5be40ebada87c8d21692036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294f1b2db5be40ebada87c8d21692036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } } }
I20260810 18:05:29.747334 18023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:29.747687 18022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "294f1b2db5be40ebada87c8d21692036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "294f1b2db5be40ebada87c8d21692036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } } }
I20260810 18:05:29.747785 18022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:29.748103 18035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:29.751696 18035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:29.753640 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:29.748325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35585 (local address 127.0.0.1:35969)
0810 18:05:29.748429 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:29.748434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:29.749308 (+ 874us) server_negotiation.cc:407] Connection header received
0810 18:05:29.749401 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:29.749426 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:29.749479 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:29.749569 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:29.749837 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:29.750598 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:29.751734 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:29.752063 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:29.752183 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:29.753151 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:29.753155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:29.753158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:29.753171 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:29.753203 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:29.753218 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:29.753298 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:29.753455 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 18:05:29.753512 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:05:29.757859 18035 catalog_manager.cc:1347] Generated new cluster ID: c1691031423c4854b9991657c46d1f62
I20260810 18:05:29.757928 18035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:29.768648 18035 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:29.769716 18035 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:29.777292 18035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 294f1b2db5be40ebada87c8d21692036: Generated new TSK 0
I20260810 18:05:29.778038 18035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:29.789628 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 17940
2026-08-10T18:05:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:29 dist-test-slave-srgv krb5kdc[18048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:29 dist-test-slave-srgv krb5kdc[18048](info): set up 2 sockets
Aug 10 18:05:29 dist-test-slave-srgv krb5kdc[18048](info): commencing operation
krb5kdc: starting...
W20260810 18:05:30.844398 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:30 dist-test-slave-srgv krb5kdc[18048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:30.976548 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:31.092133 18062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:31.092514 18062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:31.092592 18062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:31.096819 18062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:31.096956 18062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:31.096994 18062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:31.097035 18062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:31.097071 18062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:31.102823 18062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:31.104071 18062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:31.105158 18062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:31.113956 18068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:31.113957 18067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:31.113976 18070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:31.114416 18062 server_base.cc:1034] running on GCE node
I20260810 18:05:31.115073 18062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:31.116264 18062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:31.117503 18062 hybrid_clock.cc:648] HybridClock initialized: now 1786385131117473 us; error 45 us; skew 500 ppm
Aug 10 18:05:31 dist-test-slave-srgv krb5kdc[18048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:31.120501 18062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:31.121575 18062 webserver.cc:460] Webserver started at http://127.0.0.1:32797/ using document root <none> and password file <none>
I20260810 18:05:31.122180 18062 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:31.122282 18062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:31.122527 18062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:31.124727 18062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "899e4cf1711e48829803dadd715a68a6"
format_stamp: "Formatted at 2026-08-10 18:05:31 on dist-test-slave-srgv"
server_key: "0fef89f80e8c813137288e45ff18a44f"
server_key_iv: "cf12e926d340937832aa9d9b37fb79f1"
server_key_version: "encryptionkey@0"
I20260810 18:05:31.125342 18062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "899e4cf1711e48829803dadd715a68a6"
format_stamp: "Formatted at 2026-08-10 18:05:31 on dist-test-slave-srgv"
server_key: "0fef89f80e8c813137288e45ff18a44f"
server_key_iv: "cf12e926d340937832aa9d9b37fb79f1"
server_key_version: "encryptionkey@0"
I20260810 18:05:31.129755 18062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:05:31.132406 18077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:31.133778 18062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:05:31.133934 18062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "899e4cf1711e48829803dadd715a68a6"
format_stamp: "Formatted at 2026-08-10 18:05:31 on dist-test-slave-srgv"
server_key: "0fef89f80e8c813137288e45ff18a44f"
server_key_iv: "cf12e926d340937832aa9d9b37fb79f1"
server_key_version: "encryptionkey@0"
I20260810 18:05:31.134061 18062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:31.138959 18062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:31.142112 18062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:31.142328 18062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:31.149580 18062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40875
I20260810 18:05:31.149628 18129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40875 every 8 connection(s)
I20260810 18:05:31.150586 18062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:31.153401 18130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:31.154023 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 18062
I20260810 18:05:31.154209 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:31.154506 29005 external_mini_cluster.cc:1428] Setting key 25c5a3d224a6ab1b1d02a46fd5328e65
I20260810 18:05:31.160398 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:31.155898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39467 (local address 127.0.0.1:40875)
0810 18:05:31.156347 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:31.156368 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:31.156412 (+ 44us) server_negotiation.cc:407] Connection header received
0810 18:05:31.157172 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:31.157205 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:31.157625 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:31.158023 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:31.158878 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:31.158884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:31.158899 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:31.158913 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:31.158981 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:31.159010 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:31.159164 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:31.159252 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 18:05:31.159415 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":106,"threads_started":1}
I20260810 18:05:31.161360 18130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6: Bootstrap starting.
I20260810 18:05:31.163522 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:31.164070 18130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:31.164974 18130 log.cc:824] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:31.167554 18130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6: No bootstrap required, opened a new log
I20260810 18:05:31.167794 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:31.165958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49539 (local address 127.0.0.1:40875)
0810 18:05:31.166082 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:31.166086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:31.166241 (+ 155us) server_negotiation.cc:407] Connection header received
0810 18:05:31.166365 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:31.166368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:31.166399 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:31.166470 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:31.167265 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:31.167267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:31.167269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:31.167281 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:31.167318 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:31.167331 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:31.167408 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:31.167573 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 18:05:31.167665 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:05:31.171650 18130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e4cf1711e48829803dadd715a68a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260810 18:05:31.171985 18130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:31.172091 18130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899e4cf1711e48829803dadd715a68a6, State: Initialized, Role: FOLLOWER
I20260810 18:05:31.172550 18130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e4cf1711e48829803dadd715a68a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260810 18:05:31.172676 18130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:31.172765 18130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:31.172865 18130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:31.174005 18130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e4cf1711e48829803dadd715a68a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260810 18:05:31.174377 18130 leader_election.cc:302] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 899e4cf1711e48829803dadd715a68a6; no voters:
I20260810 18:05:31.174722 18130 leader_election.cc:288] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:31.174893 18141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:31.175180 18141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [term 1 LEADER]: Becoming Leader. State: Replica: 899e4cf1711e48829803dadd715a68a6, State: Running, Role: LEADER
I20260810 18:05:31.175513 18141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e4cf1711e48829803dadd715a68a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } }
I20260810 18:05:31.176043 18130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:31.177325 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "899e4cf1711e48829803dadd715a68a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e4cf1711e48829803dadd715a68a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } } }
I20260810 18:05:31.177479 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:31.177891 18143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899e4cf1711e48829803dadd715a68a6. Latest consensus state: current_term: 1 leader_uuid: "899e4cf1711e48829803dadd715a68a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e4cf1711e48829803dadd715a68a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40875 } } }
I20260810 18:05:31.178004 18143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:31.177874 18151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:31.181995 18151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:31.187930 18151 catalog_manager.cc:1347] Generated new cluster ID: 805196bcdfb34edc84d7c6fe07ce529b
I20260810 18:05:31.188027 18151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:31.209745 18151 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:31.210228 18151 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:31.220722 18151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 899e4cf1711e48829803dadd715a68a6: Generated new TSK 0
I20260810 18:05:31.221632 18151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:31.227970 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 18062
2026-08-10T18:05:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:31 dist-test-slave-srgv krb5kdc[18163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:31 dist-test-slave-srgv krb5kdc[18163](info): set up 2 sockets
Aug 10 18:05:31 dist-test-slave-srgv krb5kdc[18163](info): commencing operation
krb5kdc: starting...
W20260810 18:05:32.288810 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:32 dist-test-slave-srgv krb5kdc[18163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:32.416428 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:32.531445 18178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:32.531721 18178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:32.531765 18178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:32.535897 18178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:32.535965 18178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:32.535985 18178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:32.536003 18178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:32.536018 18178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:32.541390 18178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:32.542538 18178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:32.543522 18178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:32.552529 18186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:32.552446 18184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:32.552454 18183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:32.552690 18178 server_base.cc:1034] running on GCE node
I20260810 18:05:32.553377 18178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:32.554602 18178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:32.555812 18178 hybrid_clock.cc:648] HybridClock initialized: now 1786385132555780 us; error 49 us; skew 500 ppm
Aug 10 18:05:32 dist-test-slave-srgv krb5kdc[18163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:32.558955 18178 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:32.559966 18178 webserver.cc:460] Webserver started at http://127.0.0.1:35367/ using document root <none> and password file <none>
I20260810 18:05:32.560562 18178 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:32.560655 18178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:32.560894 18178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:32.563189 18178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "78d2cb297b4b4c81925effe191a67095"
format_stamp: "Formatted at 2026-08-10 18:05:32 on dist-test-slave-srgv"
server_key: "0870a39277a5f1e54ad01445ece2dbde"
server_key_iv: "24a6a007ee247b052fa68186052cfa07"
server_key_version: "encryptionkey@0"
I20260810 18:05:32.563783 18178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "78d2cb297b4b4c81925effe191a67095"
format_stamp: "Formatted at 2026-08-10 18:05:32 on dist-test-slave-srgv"
server_key: "0870a39277a5f1e54ad01445ece2dbde"
server_key_iv: "24a6a007ee247b052fa68186052cfa07"
server_key_version: "encryptionkey@0"
I20260810 18:05:32.567732 18178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:32.570371 18193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:32.571938 18178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 18:05:32.572103 18178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "78d2cb297b4b4c81925effe191a67095"
format_stamp: "Formatted at 2026-08-10 18:05:32 on dist-test-slave-srgv"
server_key: "0870a39277a5f1e54ad01445ece2dbde"
server_key_iv: "24a6a007ee247b052fa68186052cfa07"
server_key_version: "encryptionkey@0"
I20260810 18:05:32.572249 18178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:32.582065 18178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:32.585628 18178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:32.585848 18178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:32.593473 18178 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46607
I20260810 18:05:32.593494 18246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46607 every 8 connection(s)
I20260810 18:05:32.594491 18178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:32.597296 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:32.603227 18247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095: Bootstrap starting.
I20260810 18:05:32.603641 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 18178
I20260810 18:05:32.603790 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:32.604079 29005 external_mini_cluster.cc:1428] Setting key 225a89b85d8fdbcf60fa3e6fc6c8f1f4
I20260810 18:05:32.606169 18247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:32.607101 18247 log.cc:824] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:32.609531 18247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095: No bootstrap required, opened a new log
Aug 10 18:05:32 dist-test-slave-srgv krb5kdc[18163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:05:32.613891 18247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d2cb297b4b4c81925effe191a67095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46607 } }
I20260810 18:05:32.614163 18247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:32.614238 18247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d2cb297b4b4c81925effe191a67095, State: Initialized, Role: FOLLOWER
I20260810 18:05:32.614759 18247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d2cb297b4b4c81925effe191a67095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46607 } }
I20260810 18:05:32.614930 18247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:32.615010 18247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:32.615127 18247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:32.616569 18247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d2cb297b4b4c81925effe191a67095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46607 } }
I20260810 18:05:32.616981 18247 leader_election.cc:302] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d2cb297b4b4c81925effe191a67095; no voters:
I20260810 18:05:32.617355 18247 leader_election.cc:288] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:32.617520 18252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:32.617779 18252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [term 1 LEADER]: Becoming Leader. State: Replica: 78d2cb297b4b4c81925effe191a67095, State: Running, Role: LEADER
I20260810 18:05:32.618132 18252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d2cb297b4b4c81925effe191a67095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46607 } }
I20260810 18:05:32.618919 18247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:32.619190 18253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d2cb297b4b4c81925effe191a67095. Latest consensus state: current_term: 1 leader_uuid: "78d2cb297b4b4c81925effe191a67095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d2cb297b4b4c81925effe191a67095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46607 } } }
I20260810 18:05:32.619388 18253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:32.619747 18254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d2cb297b4b4c81925effe191a67095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d2cb297b4b4c81925effe191a67095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46607 } } }
I20260810 18:05:32.619876 18254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:32.619822 18256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:32.622547 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:32.605599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:52815 (local address 127.0.0.1:46607)
0810 18:05:32.606069 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:32.606092 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:32.606152 (+ 60us) server_negotiation.cc:407] Connection header received
0810 18:05:32.607090 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:32.607118 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:32.607601 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:32.607900 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:32.608708 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:32.610198 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:32.611051 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:32.611797 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:32.611957 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:32.614805 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:32.614812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:32.614829 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:32.614865 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:32.616996 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:32.618880 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:32.618892 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:32.618898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:32.619122 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:32.619584 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:32.619590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:32.619594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:32.620187 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:32.620420 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:32.621245 (+ 825us) server_negotiation.cc:299] Negotiation successful
0810 18:05:32.621520 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":139,"threads_started":1}
I20260810 18:05:32.623540 18256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:32.625962 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:32.629935 18256 catalog_manager.cc:1347] Generated new cluster ID: d7a2caad9f5d44709972b8adabe31f0c
I20260810 18:05:32.630002 18256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:32.640893 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:32.628510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:41177 (local address 127.0.0.1:46607)
0810 18:05:32.628608 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:32.628612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:32.628964 (+ 352us) server_negotiation.cc:407] Connection header received
0810 18:05:32.629123 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:32.629126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:32.629180 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:32.629270 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:32.634196 (+ 4926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:32.634773 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:32.635667 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:32.635901 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:32.636007 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:32.637553 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:32.637563 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:32.637565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:32.637593 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:32.639071 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:32.639571 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:32.639580 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:32.639581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:32.639656 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:32.639992 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:32.639995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:32.639997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:32.640192 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:32.640365 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:32.640627 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 18:05:32.640727 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 18:05:32.656805 18256 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:32.657862 18256 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:32.662361 18256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d2cb297b4b4c81925effe191a67095: Generated new TSK 0
I20260810 18:05:32.663203 18256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:32.671969 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 18178
W20260810 18:05:32.678406 18161 connection.cc:535] client connection to 127.0.0.1:46607 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46607): Connection reset by peer (error 104)
2026-08-10T18:05:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:32 dist-test-slave-srgv krb5kdc[18279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:32 dist-test-slave-srgv krb5kdc[18279](info): set up 2 sockets
Aug 10 18:05:32 dist-test-slave-srgv krb5kdc[18279](info): commencing operation
krb5kdc: starting...
W20260810 18:05:33.749040 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:33 dist-test-slave-srgv krb5kdc[18279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:33.880075 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:33.996099 18294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:33.996436 18294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:33.996523 18294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:34.000639 18294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:34.000746 18294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:34.000784 18294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:34.000837 18294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:34.000872 18294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:34.006474 18294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:34.007634 18294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:34.008701 18294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:34.017522 18299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:34.017586 18294 server_base.cc:1034] running on GCE node
W20260810 18:05:34.017522 18300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:34.017527 18302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:34.018443 18294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:34.019613 18294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:34.020836 18294 hybrid_clock.cc:648] HybridClock initialized: now 1786385134020818 us; error 33 us; skew 500 ppm
Aug 10 18:05:34 dist-test-slave-srgv krb5kdc[18279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:34.027115 18294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:34.028108 18294 webserver.cc:460] Webserver started at http://127.0.0.1:36877/ using document root <none> and password file <none>
I20260810 18:05:34.028650 18294 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:34.028733 18294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:34.029031 18294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:34.031391 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "862a7333ab0b4637a92246a9407a1700"
format_stamp: "Formatted at 2026-08-10 18:05:34 on dist-test-slave-srgv"
server_key: "3233af26099bc2aafefbf06cf39d69d3"
server_key_iv: "356e3fc26744691faa0f6c307b050ac5"
server_key_version: "encryptionkey@0"
I20260810 18:05:34.032016 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "862a7333ab0b4637a92246a9407a1700"
format_stamp: "Formatted at 2026-08-10 18:05:34 on dist-test-slave-srgv"
server_key: "3233af26099bc2aafefbf06cf39d69d3"
server_key_iv: "356e3fc26744691faa0f6c307b050ac5"
server_key_version: "encryptionkey@0"
I20260810 18:05:34.036103 18294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 18:05:34.039047 18310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:34.040449 18294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:05:34.040589 18294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "862a7333ab0b4637a92246a9407a1700"
format_stamp: "Formatted at 2026-08-10 18:05:34 on dist-test-slave-srgv"
server_key: "3233af26099bc2aafefbf06cf39d69d3"
server_key_iv: "356e3fc26744691faa0f6c307b050ac5"
server_key_version: "encryptionkey@0"
I20260810 18:05:34.040719 18294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:34.058734 18294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:34.062808 18294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:34.063026 18294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:34.070508 18294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33151
I20260810 18:05:34.070538 18363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33151 every 8 connection(s)
I20260810 18:05:34.071502 18294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:34.074432 18364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:34.077584 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 18294
I20260810 18:05:34.077755 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:34.078081 29005 external_mini_cluster.cc:1428] Setting key 1819850c23b1e880d4d1da46d9b743f9
I20260810 18:05:34.081250 18364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700: Bootstrap starting.
I20260810 18:05:34.084014 18364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:34.084908 18364 log.cc:824] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:34.087316 18364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700: No bootstrap required, opened a new log
I20260810 18:05:34.087779 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:34.079594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:55387 (local address 127.0.0.1:33151)
0810 18:05:34.080080 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:34.080107 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:34.080143 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:05:34.080783 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:34.080800 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:34.081146 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:34.081534 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:34.081906 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:34.082895 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:34.084252 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:34.084758 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:34.084881 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:34.085945 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:34.085951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:34.085964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:34.086367 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:34.086420 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:34.086443 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:34.086547 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:34.086775 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 18:05:34.086949 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":147,"threads_started":1}
I20260810 18:05:34.090894 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:34.091642 18364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a7333ab0b4637a92246a9407a1700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260810 18:05:34.091871 18364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:34.091926 18364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862a7333ab0b4637a92246a9407a1700, State: Initialized, Role: FOLLOWER
I20260810 18:05:34.092427 18364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a7333ab0b4637a92246a9407a1700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260810 18:05:34.092552 18364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:34.092634 18364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:34.092736 18364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:34.094159 18364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a7333ab0b4637a92246a9407a1700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260810 18:05:34.094539 18364 leader_election.cc:302] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 862a7333ab0b4637a92246a9407a1700; no voters:
I20260810 18:05:34.094877 18364 leader_election.cc:288] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:34.094987 18374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:34.095259 18374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [term 1 LEADER]: Becoming Leader. State: Replica: 862a7333ab0b4637a92246a9407a1700, State: Running, Role: LEADER
I20260810 18:05:34.095646 18374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a7333ab0b4637a92246a9407a1700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } }
I20260810 18:05:34.096252 18364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:34.096900 18374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 862a7333ab0b4637a92246a9407a1700. Latest consensus state: current_term: 1 leader_uuid: "862a7333ab0b4637a92246a9407a1700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a7333ab0b4637a92246a9407a1700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } } }
I20260810 18:05:34.097034 18374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:34.097352 18375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "862a7333ab0b4637a92246a9407a1700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862a7333ab0b4637a92246a9407a1700" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33151 } } }
I20260810 18:05:34.097544 18375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:34.099117 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:34.093125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:40579 (local address 127.0.0.1:33151)
0810 18:05:34.093240 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:34.093244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:34.093690 (+ 446us) server_negotiation.cc:407] Connection header received
0810 18:05:34.093817 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:34.093819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:34.093861 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:34.093928 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:34.094292 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:34.094927 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:34.096097 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:34.096470 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:34.096567 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:34.098067 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:34.098072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:34.098077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:34.098095 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:34.098130 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:34.098146 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:34.098751 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:34.098812 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 18:05:34.098993 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 18:05:34.102090 18389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:34.102182 18389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:05:34.102269 18379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:34.102974 18379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:34.109884 18379 catalog_manager.cc:1347] Generated new cluster ID: d6bd49df510648b78a0daa8f6a2201bc
I20260810 18:05:34.109977 18379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:34.127969 18379 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:34.129356 18379 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:34.136314 18379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 862a7333ab0b4637a92246a9407a1700: Generated new TSK 0
I20260810 18:05:34.137063 18379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:34.141484 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 18294
W20260810 18:05:34.148249 18277 connection.cc:535] client connection to 127.0.0.1:33151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33151): Connection reset by peer (error 104)
2026-08-10T18:05:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:34 dist-test-slave-srgv krb5kdc[18395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:34 dist-test-slave-srgv krb5kdc[18395](info): set up 2 sockets
Aug 10 18:05:34 dist-test-slave-srgv krb5kdc[18395](info): commencing operation
krb5kdc: starting...
W20260810 18:05:35.197024 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:35 dist-test-slave-srgv krb5kdc[18395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:35.328123 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:35.444600 18408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:35.444953 18408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:35.445034 18408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:35.449183 18408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:35.449329 18408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:35.449373 18408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:35.449463 18408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:35.449503 18408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:35.455188 18408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:35.456411 18408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:35.457561 18408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:35.467437 18414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:35.467465 18413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:35.467569 18416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:35.468101 18408 server_base.cc:1034] running on GCE node
I20260810 18:05:35.468932 18408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:35.470693 18408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:35.471887 18408 hybrid_clock.cc:648] HybridClock initialized: now 1786385135471850 us; error 67 us; skew 500 ppm
Aug 10 18:05:35 dist-test-slave-srgv krb5kdc[18395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:35.475934 18408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:35.476930 18408 webserver.cc:460] Webserver started at http://127.0.0.1:44905/ using document root <none> and password file <none>
I20260810 18:05:35.477555 18408 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:35.477698 18408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:35.477946 18408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:35.480091 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3"
format_stamp: "Formatted at 2026-08-10 18:05:35 on dist-test-slave-srgv"
server_key: "dd5e9bb05ab45b6a80e97ccb0b493c41"
server_key_iv: "2eb8b0f5ae2c2313e450a8802d7804e1"
server_key_version: "encryptionkey@0"
I20260810 18:05:35.480696 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3"
format_stamp: "Formatted at 2026-08-10 18:05:35 on dist-test-slave-srgv"
server_key: "dd5e9bb05ab45b6a80e97ccb0b493c41"
server_key_iv: "2eb8b0f5ae2c2313e450a8802d7804e1"
server_key_version: "encryptionkey@0"
I20260810 18:05:35.485535 18408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 18:05:35.488373 18424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:35.490033 18408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:35.490187 18408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3"
format_stamp: "Formatted at 2026-08-10 18:05:35 on dist-test-slave-srgv"
server_key: "dd5e9bb05ab45b6a80e97ccb0b493c41"
server_key_iv: "2eb8b0f5ae2c2313e450a8802d7804e1"
server_key_version: "encryptionkey@0"
I20260810 18:05:35.490312 18408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:35.497038 18408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:35.501035 18408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:35.501284 18408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:35.509114 18408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40519
I20260810 18:05:35.509179 18480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40519 every 8 connection(s)
I20260810 18:05:35.510221 18408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:35.514389 18481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:35.516202 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 18408
I20260810 18:05:35.516341 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:35.516690 29005 external_mini_cluster.cc:1428] Setting key f774b19a709e7140aac356e12163166b
I20260810 18:05:35.521391 18481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3: Bootstrap starting.
I20260810 18:05:35.522485 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:35.518253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:33523 (local address 127.0.0.1:40519)
0810 18:05:35.518901 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:35.518921 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:35.518962 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:05:35.519654 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:35.519673 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:35.519961 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:35.520338 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:35.521180 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:35.521186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:35.521205 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:35.521215 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:35.521270 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:35.521289 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:35.521473 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:35.521560 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 18:05:35.521728 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":138,"threads_started":1}
I20260810 18:05:35.523866 18481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:35.524752 18481 log.cc:824] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:35.525281 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:35.526965 18481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3: No bootstrap required, opened a new log
I20260810 18:05:35.529222 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:35.527458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:45585 (local address 127.0.0.1:40519)
0810 18:05:35.527601 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:35.527606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:35.527834 (+ 228us) server_negotiation.cc:407] Connection header received
0810 18:05:35.527964 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:35.527966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:35.527995 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:35.528061 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:35.528817 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:35.528820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:35.528822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:35.528832 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:35.528859 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:35.528869 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:35.528951 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:35.529051 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 18:05:35.529084 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:05:35.531525 18481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40519 } }
I20260810 18:05:35.531765 18481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:35.531800 18481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97f5a92d4f40e1ad413b5e9e2d17c3, State: Initialized, Role: FOLLOWER
I20260810 18:05:35.532213 18481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40519 } }
I20260810 18:05:35.532341 18481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:35.532382 18481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:35.532451 18481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:35.533358 18481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40519 } }
I20260810 18:05:35.533723 18481 leader_election.cc:302] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d97f5a92d4f40e1ad413b5e9e2d17c3; no voters:
I20260810 18:05:35.533989 18481 leader_election.cc:288] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:35.534121 18491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:35.534380 18491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d97f5a92d4f40e1ad413b5e9e2d17c3, State: Running, Role: LEADER
I20260810 18:05:35.534693 18491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40519 } }
I20260810 18:05:35.535377 18481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:35.536473 18492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40519 } } }
I20260810 18:05:35.536625 18492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:35.536906 18493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d97f5a92d4f40e1ad413b5e9e2d17c3. Latest consensus state: current_term: 1 leader_uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97f5a92d4f40e1ad413b5e9e2d17c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40519 } } }
I20260810 18:05:35.537014 18493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:35.537091 18500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:35.540253 18500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:35.545871 18500 catalog_manager.cc:1347] Generated new cluster ID: 25d46a41a03648c69b6d96544dc84337
I20260810 18:05:35.545956 18500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:35.558287 18500 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:35.558701 18500 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:35.568039 18500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d97f5a92d4f40e1ad413b5e9e2d17c3: Generated new TSK 0
I20260810 18:05:35.568759 18500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:35.572665 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 18408
2026-08-10T18:05:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:35 dist-test-slave-srgv krb5kdc[18512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:35 dist-test-slave-srgv krb5kdc[18512](info): set up 2 sockets
Aug 10 18:05:35 dist-test-slave-srgv krb5kdc[18512](info): commencing operation
krb5kdc: starting...
W20260810 18:05:36.636435 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:36 dist-test-slave-srgv krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:36.764585 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:36.879346 18525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:36.879647 18525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:36.879690 18525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:36.883783 18525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:36.883864 18525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:36.883884 18525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:36.883901 18525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:36.883916 18525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:36.889338 18525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:36.890506 18525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:36.891590 18525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:36.900431 18530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:36.900425 18533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:36.900425 18531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:36.900537 18525 server_base.cc:1034] running on GCE node
I20260810 18:05:36.901305 18525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:36.902627 18525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:36.903831 18525 hybrid_clock.cc:648] HybridClock initialized: now 1786385136903816 us; error 41 us; skew 500 ppm
Aug 10 18:05:36 dist-test-slave-srgv krb5kdc[18512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:36.907021 18525 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:36.908082 18525 webserver.cc:460] Webserver started at http://127.0.0.1:43513/ using document root <none> and password file <none>
I20260810 18:05:36.908690 18525 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:36.908787 18525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:36.909039 18525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:36.911306 18525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "4aa93ef54cad4bcc9db340bedbce53ed"
format_stamp: "Formatted at 2026-08-10 18:05:36 on dist-test-slave-srgv"
server_key: "5887c17a472902b45d3749b6860941e1"
server_key_iv: "620e1bfbbdeb48eb6ab4b03ac83746ab"
server_key_version: "encryptionkey@0"
I20260810 18:05:36.911916 18525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "4aa93ef54cad4bcc9db340bedbce53ed"
format_stamp: "Formatted at 2026-08-10 18:05:36 on dist-test-slave-srgv"
server_key: "5887c17a472902b45d3749b6860941e1"
server_key_iv: "620e1bfbbdeb48eb6ab4b03ac83746ab"
server_key_version: "encryptionkey@0"
I20260810 18:05:36.915990 18525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:36.918676 18540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:36.919935 18525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:05:36.920118 18525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "4aa93ef54cad4bcc9db340bedbce53ed"
format_stamp: "Formatted at 2026-08-10 18:05:36 on dist-test-slave-srgv"
server_key: "5887c17a472902b45d3749b6860941e1"
server_key_iv: "620e1bfbbdeb48eb6ab4b03ac83746ab"
server_key_version: "encryptionkey@0"
I20260810 18:05:36.920243 18525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:36.935775 18525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:36.938923 18525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:36.939154 18525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:36.946704 18525 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37285
I20260810 18:05:36.946732 18596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37285 every 8 connection(s)
I20260810 18:05:36.947686 18525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:36.950474 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:36.951901 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 18525
I20260810 18:05:36.952049 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:36.952381 29005 external_mini_cluster.cc:1428] Setting key 72adeb506d03289e771d639cac236bcb
I20260810 18:05:36.956943 18597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed: Bootstrap starting.
I20260810 18:05:36.959546 18597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:36.960493 18597 log.cc:824] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:05:36 dist-test-slave-srgv krb5kdc[18512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:05:36.962687 18597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed: No bootstrap required, opened a new log
I20260810 18:05:36.966442 18597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa93ef54cad4bcc9db340bedbce53ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260810 18:05:36.966786 18597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:36.966874 18597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa93ef54cad4bcc9db340bedbce53ed, State: Initialized, Role: FOLLOWER
I20260810 18:05:36.967361 18597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa93ef54cad4bcc9db340bedbce53ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260810 18:05:36.967510 18597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:36.967517 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:36.953963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:47863 (local address 127.0.0.1:37285)
0810 18:05:36.954401 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:36.954419 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:36.954455 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:05:36.955105 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:36.955122 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:36.955476 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:36.955763 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:36.956704 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:36.957821 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:36.959021 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:36.959508 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:36.959659 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:36.962575 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:36.962580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:36.962594 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:36.962623 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:36.964395 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:36.964929 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:36.964934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:36.964938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:36.965046 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:36.965380 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:36.965388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:36.965390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:36.965841 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:36.966080 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:36.966402 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 18:05:36.966630 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":124,"threads_started":1}
I20260810 18:05:36.967604 18597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:36.967718 18597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:36.968752 18597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa93ef54cad4bcc9db340bedbce53ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260810 18:05:36.969105 18597 leader_election.cc:302] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aa93ef54cad4bcc9db340bedbce53ed; no voters:
I20260810 18:05:36.969499 18597 leader_election.cc:288] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:36.969635 18602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:36.969895 18602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [term 1 LEADER]: Becoming Leader. State: Replica: 4aa93ef54cad4bcc9db340bedbce53ed, State: Running, Role: LEADER
I20260810 18:05:36.970274 18602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa93ef54cad4bcc9db340bedbce53ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260810 18:05:36.970734 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:36.970819 18597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:36.973310 18604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa93ef54cad4bcc9db340bedbce53ed. Latest consensus state: current_term: 1 leader_uuid: "4aa93ef54cad4bcc9db340bedbce53ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa93ef54cad4bcc9db340bedbce53ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } } }
I20260810 18:05:36.973570 18603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aa93ef54cad4bcc9db340bedbce53ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa93ef54cad4bcc9db340bedbce53ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } } }
I20260810 18:05:36.973722 18604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:36.974038 18603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed [sys.catalog]: This master's current role is: LEADER
W20260810 18:05:36.976517 18620 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:36.976613 18620 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:05:36.976678 18624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:36.977581 18624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:36.981804 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:36.973616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:59785 (local address 127.0.0.1:37285)
0810 18:05:36.973737 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:36.973744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:36.974141 (+ 397us) server_negotiation.cc:407] Connection header received
0810 18:05:36.974337 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:36.974340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:36.974387 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:36.974477 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:36.975140 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:36.975655 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:36.976470 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:36.976782 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:36.976886 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:36.978446 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:36.978450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:36.978453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:36.978478 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:36.979957 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:36.980446 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:36.980450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:36.980452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:36.980533 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:36.980857 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:36.980860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:36.980862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:36.981037 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:36.981189 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:36.981505 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 18:05:36.981609 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:05:36.984593 18624 catalog_manager.cc:1347] Generated new cluster ID: f5bc553658494dd5ac6fe6f7a18a672b
I20260810 18:05:36.984668 18624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:37.003832 18624 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:37.004894 18624 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:37.011346 18624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aa93ef54cad4bcc9db340bedbce53ed: Generated new TSK 0
I20260810 18:05:37.012070 18624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:37.022712 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 18525
W20260810 18:05:37.029337 18510 connection.cc:535] client connection to 127.0.0.1:37285 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37285): Connection reset by peer (error 104)
2026-08-10T18:05:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:37 dist-test-slave-srgv krb5kdc[18629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:37 dist-test-slave-srgv krb5kdc[18629](info): set up 2 sockets
Aug 10 18:05:37 dist-test-slave-srgv krb5kdc[18629](info): commencing operation
krb5kdc: starting...
W20260810 18:05:38.080619 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:38 dist-test-slave-srgv krb5kdc[18629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:38.212486 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:38.330277 18643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:38.330616 18643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:38.330711 18643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:38.334884 18643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:38.334981 18643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:38.335021 18643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:38.335078 18643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:38.335119 18643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:38.340740 18643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:38.342001 18643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:38.343081 18643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:38.351919 18649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:38.351915 18652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:38.351936 18650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:38.352186 18643 server_base.cc:1034] running on GCE node
I20260810 18:05:38.352870 18643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:38.354122 18643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:38.355337 18643 hybrid_clock.cc:648] HybridClock initialized: now 1786385138355310 us; error 52 us; skew 500 ppm
Aug 10 18:05:38 dist-test-slave-srgv krb5kdc[18629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:38.358415 18643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:38.359403 18643 webserver.cc:460] Webserver started at http://127.0.0.1:42713/ using document root <none> and password file <none>
I20260810 18:05:38.359977 18643 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:38.360073 18643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:38.360376 18643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:38.362574 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "aa1fd14495c14eb6ba3efd205de402d5"
format_stamp: "Formatted at 2026-08-10 18:05:38 on dist-test-slave-srgv"
server_key: "8e998aa6a42af6454d116719cef23f63"
server_key_iv: "51a26cebc6b875d2d315bdf62653789e"
server_key_version: "encryptionkey@0"
I20260810 18:05:38.363163 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "aa1fd14495c14eb6ba3efd205de402d5"
format_stamp: "Formatted at 2026-08-10 18:05:38 on dist-test-slave-srgv"
server_key: "8e998aa6a42af6454d116719cef23f63"
server_key_iv: "51a26cebc6b875d2d315bdf62653789e"
server_key_version: "encryptionkey@0"
I20260810 18:05:38.366979 18643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:38.369599 18659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:38.370853 18643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:05:38.371001 18643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "aa1fd14495c14eb6ba3efd205de402d5"
format_stamp: "Formatted at 2026-08-10 18:05:38 on dist-test-slave-srgv"
server_key: "8e998aa6a42af6454d116719cef23f63"
server_key_iv: "51a26cebc6b875d2d315bdf62653789e"
server_key_version: "encryptionkey@0"
I20260810 18:05:38.371145 18643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:38.388644 18643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:38.391851 18643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:38.392094 18643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:38.400013 18643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44665
I20260810 18:05:38.400029 18715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44665 every 8 connection(s)
I20260810 18:05:38.401033 18643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:38.403947 18716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:38.409744 18716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5: Bootstrap starting.
I20260810 18:05:38.410189 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 18643
I20260810 18:05:38.410349 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:38.410617 29005 external_mini_cluster.cc:1428] Setting key a4b3a08c8e00dc6f673b4d33e4d81549
I20260810 18:05:38.412401 18716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:38.413129 18716 log.cc:824] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:38.415071 18716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5: No bootstrap required, opened a new log
I20260810 18:05:38.418731 18716 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1fd14495c14eb6ba3efd205de402d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44665 } }
I20260810 18:05:38.419014 18716 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:38.419131 18716 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1fd14495c14eb6ba3efd205de402d5, State: Initialized, Role: FOLLOWER
I20260810 18:05:38.419606 18716 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1fd14495c14eb6ba3efd205de402d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44665 } }
I20260810 18:05:38.419759 18716 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:38.419842 18716 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:38.419860 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:38.412030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:43201 (local address 127.0.0.1:44665)
0810 18:05:38.412446 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:38.412462 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:38.412496 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:05:38.413186 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:38.413202 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:38.413638 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:38.413960 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:38.414382 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:38.415442 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:38.416352 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:38.416834 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:38.417007 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:38.418021 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:38.418027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:38.418039 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:38.418432 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:38.418485 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:38.418508 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:38.418648 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:38.418843 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 18:05:38.419003 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":130,"threads_started":1}
I20260810 18:05:38.419970 18716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:38.420872 18716 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1fd14495c14eb6ba3efd205de402d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44665 } }
I20260810 18:05:38.421206 18716 leader_election.cc:302] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa1fd14495c14eb6ba3efd205de402d5; no voters:
I20260810 18:05:38.421581 18716 leader_election.cc:288] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:38.421666 18722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:38.421927 18722 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [term 1 LEADER]: Becoming Leader. State: Replica: aa1fd14495c14eb6ba3efd205de402d5, State: Running, Role: LEADER
I20260810 18:05:38.422276 18722 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1fd14495c14eb6ba3efd205de402d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44665 } }
I20260810 18:05:38.422708 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:38.422930 18716 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:38.424106 18723 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa1fd14495c14eb6ba3efd205de402d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1fd14495c14eb6ba3efd205de402d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44665 } } }
I20260810 18:05:38.424252 18723 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:38.424099 18724 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa1fd14495c14eb6ba3efd205de402d5. Latest consensus state: current_term: 1 leader_uuid: "aa1fd14495c14eb6ba3efd205de402d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa1fd14495c14eb6ba3efd205de402d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44665 } } }
I20260810 18:05:38.424631 18724 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:38.424998 18735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:38.428766 18735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:38.430413 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:38.425643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:41289 (local address 127.0.0.1:44665)
0810 18:05:38.425783 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:38.425788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:38.426031 (+ 243us) server_negotiation.cc:407] Connection header received
0810 18:05:38.426140 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:38.426143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:38.426194 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:38.426275 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:38.426573 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:38.427398 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:38.428413 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:38.428793 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:38.428894 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:38.429961 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:38.429965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:38.429967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:38.429981 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:38.430012 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:38.430023 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:38.430137 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:38.430242 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 18:05:38.430274 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 18:05:38.434676 18735 catalog_manager.cc:1347] Generated new cluster ID: 362c09ecd12948ec9c502611c41a95be
I20260810 18:05:38.434751 18735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:38.460125 18735 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:38.461076 18735 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:38.467772 18735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa1fd14495c14eb6ba3efd205de402d5: Generated new TSK 0
I20260810 18:05:38.468482 18735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:38.474871 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 18643
W20260810 18:05:38.481072 18627 connection.cc:535] client connection to 127.0.0.1:44665 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44665): Connection reset by peer (error 104)
2026-08-10T18:05:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:38 dist-test-slave-srgv krb5kdc[18749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:38 dist-test-slave-srgv krb5kdc[18749](info): set up 2 sockets
Aug 10 18:05:38 dist-test-slave-srgv krb5kdc[18749](info): commencing operation
krb5kdc: starting...
W20260810 18:05:39.530519 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:39 dist-test-slave-srgv krb5kdc[18749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:05:39.664570 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:39.778148 18762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:39.778445 18762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:39.778489 18762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:39.782624 18762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:39.782697 18762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:39.782717 18762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:39.782734 18762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:39.782750 18762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:39.788272 18762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45223
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:39.789450 18762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:39.790482 18762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:39.799381 18768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:39.799371 18770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:39.799369 18767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:39.799799 18762 server_base.cc:1034] running on GCE node
I20260810 18:05:39.800457 18762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:39.801851 18762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:39.803017 18762 hybrid_clock.cc:648] HybridClock initialized: now 1786385139803008 us; error 41 us; skew 500 ppm
Aug 10 18:05:39 dist-test-slave-srgv krb5kdc[18749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:39.811764 18762 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:05:39.812753 18762 webserver.cc:460] Webserver started at http://127.0.0.1:39831/ using document root <none> and password file <none>
I20260810 18:05:39.813354 18762 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:39.813513 18762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:39.813755 18762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:39.815884 18762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "061c69623216440a9992956764140684"
format_stamp: "Formatted at 2026-08-10 18:05:39 on dist-test-slave-srgv"
server_key: "9984355a86aab59201edefb2ffa992a4"
server_key_iv: "bb2b9f287d110df2183169de235cc70b"
server_key_version: "encryptionkey@0"
I20260810 18:05:39.816452 18762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "061c69623216440a9992956764140684"
format_stamp: "Formatted at 2026-08-10 18:05:39 on dist-test-slave-srgv"
server_key: "9984355a86aab59201edefb2ffa992a4"
server_key_iv: "bb2b9f287d110df2183169de235cc70b"
server_key_version: "encryptionkey@0"
I20260810 18:05:39.820209 18762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 18:05:39.822822 18778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:39.824169 18762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:05:39.824324 18762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "061c69623216440a9992956764140684"
format_stamp: "Formatted at 2026-08-10 18:05:39 on dist-test-slave-srgv"
server_key: "9984355a86aab59201edefb2ffa992a4"
server_key_iv: "bb2b9f287d110df2183169de235cc70b"
server_key_version: "encryptionkey@0"
I20260810 18:05:39.824450 18762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:39.829316 18762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:39.832832 18762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:39.833040 18762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:39.840790 18762 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45223
I20260810 18:05:39.840845 18834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45223 every 8 connection(s)
I20260810 18:05:39.841885 18762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:39.844939 18835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:39.851042 18835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684: Bootstrap starting.
I20260810 18:05:39.851850 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 18762
I20260810 18:05:39.851998 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:39.852264 29005 external_mini_cluster.cc:1428] Setting key b3ae1f70ac809fb82bc7c598d583b88e
I20260810 18:05:39.854178 18835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:39.854996 18835 log.cc:824] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:39.857084 18835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684: No bootstrap required, opened a new log
I20260810 18:05:39.859062 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:39.853598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:60923 (local address 127.0.0.1:45223)
0810 18:05:39.854082 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:39.854098 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:39.854132 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:05:39.854832 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:39.854849 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:39.855167 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:39.855503 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:39.857600 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:39.857605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:39.857619 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:39.857633 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:39.857703 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:39.857731 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:39.857861 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:39.857966 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 18:05:39.858126 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":155,"threads_started":1}
I20260810 18:05:39.861675 18835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061c69623216440a9992956764140684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } }
I20260810 18:05:39.861968 18835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:39.861970 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:39.862052 18835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061c69623216440a9992956764140684, State: Initialized, Role: FOLLOWER
I20260810 18:05:39.862498 18835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061c69623216440a9992956764140684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } }
I20260810 18:05:39.862640 18835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:39.862710 18835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:39.862819 18835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:39.863878 18835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061c69623216440a9992956764140684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } }
I20260810 18:05:39.864243 18835 leader_election.cc:302] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 061c69623216440a9992956764140684; no voters:
I20260810 18:05:39.864629 18835 leader_election.cc:288] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:39.864742 18845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:39.865064 18845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [term 1 LEADER]: Becoming Leader. State: Replica: 061c69623216440a9992956764140684, State: Running, Role: LEADER
I20260810 18:05:39.865741 18845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061c69623216440a9992956764140684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } }
I20260810 18:05:39.866024 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:39.864277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:51793 (local address 127.0.0.1:45223)
0810 18:05:39.864414 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:39.864418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:39.864588 (+ 170us) server_negotiation.cc:407] Connection header received
0810 18:05:39.864701 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:39.864705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:39.864747 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:39.864836 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:39.865618 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:39.865621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:39.865624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:39.865636 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:39.865664 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:39.865677 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:39.865766 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:39.865873 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 18:05:39.865911 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:05:39.865999 18835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:39.867568 18847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 061c69623216440a9992956764140684. Latest consensus state: current_term: 1 leader_uuid: "061c69623216440a9992956764140684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061c69623216440a9992956764140684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } } }
I20260810 18:05:39.867538 18846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "061c69623216440a9992956764140684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061c69623216440a9992956764140684" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } } }
I20260810 18:05:39.867794 18846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:39.868072 18847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:39.868152 18858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:39.870870 18858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:39.876345 18858 catalog_manager.cc:1347] Generated new cluster ID: cbbc257816a74d78a62242014081e18f
I20260810 18:05:39.876415 18858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:39.889757 18858 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:39.890102 18858 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:39.898631 18858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 061c69623216440a9992956764140684: Generated new TSK 0
I20260810 18:05:39.899346 18858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:39.904460 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 18762
2026-08-10T18:05:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:39 dist-test-slave-srgv krb5kdc[18867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:39 dist-test-slave-srgv krb5kdc[18867](info): set up 2 sockets
Aug 10 18:05:39 dist-test-slave-srgv krb5kdc[18867](info): commencing operation
krb5kdc: starting...
W20260810 18:05:40.960569 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:41 dist-test-slave-srgv krb5kdc[18867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:41.093175 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37271
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:41.208887 18881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:41.209179 18881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:41.209280 18881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:41.213436 18881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:41.213507 18881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:41.213527 18881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:41.213544 18881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:41.213608 18881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:41.219144 18881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:37271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:41.220340 18881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:41.221437 18881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:41.230108 18889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:41.230127 18886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:41.230208 18881 server_base.cc:1034] running on GCE node
W20260810 18:05:41.230111 18887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:41.231024 18881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:41.232189 18881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:41.233353 18881 hybrid_clock.cc:648] HybridClock initialized: now 1786385141233330 us; error 37 us; skew 500 ppm
Aug 10 18:05:41 dist-test-slave-srgv krb5kdc[18867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385141, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:41.236363 18881 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:41.237358 18881 webserver.cc:460] Webserver started at http://127.28.83.126:36515/ using document root <none> and password file <none>
I20260810 18:05:41.238065 18881 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:41.238154 18881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:41.238384 18881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:41.240527 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "b231238800b24ccd82382fbfddbec1de"
format_stamp: "Formatted at 2026-08-10 18:05:41 on dist-test-slave-srgv"
server_key: "408bfe683c126902e9faae37b941c8de"
server_key_iv: "bdb285f1e6f4e75b5c669c74d695010b"
server_key_version: "encryptionkey@0"
I20260810 18:05:41.241139 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "b231238800b24ccd82382fbfddbec1de"
format_stamp: "Formatted at 2026-08-10 18:05:41 on dist-test-slave-srgv"
server_key: "408bfe683c126902e9faae37b941c8de"
server_key_iv: "bdb285f1e6f4e75b5c669c74d695010b"
server_key_version: "encryptionkey@0"
I20260810 18:05:41.245108 18881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:05:41.247821 18896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:41.249173 18881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 18:05:41.249351 18881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "b231238800b24ccd82382fbfddbec1de"
format_stamp: "Formatted at 2026-08-10 18:05:41 on dist-test-slave-srgv"
server_key: "408bfe683c126902e9faae37b941c8de"
server_key_iv: "bdb285f1e6f4e75b5c669c74d695010b"
server_key_version: "encryptionkey@0"
I20260810 18:05:41.249531 18881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:41.268239 18881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:41.272387 18881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:41.272568 18881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:41.280128 18881 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:37271
I20260810 18:05:41.280155 18951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:37271 every 8 connection(s)
I20260810 18:05:41.281139 18881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:41.283938 18952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:41.289748 18952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de: Bootstrap starting.
I20260810 18:05:41.290870 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 18881
I20260810 18:05:41.291029 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:41.291347 29005 external_mini_cluster.cc:1428] Setting key 6aa1d44216384328c3d0841d936be2f4
I20260810 18:05:41.292654 18952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:41.293637 18952 log.cc:824] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:41.296006 18952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de: No bootstrap required, opened a new log
I20260810 18:05:41.300207 18952 raft_consensus.cc:348] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b231238800b24ccd82382fbfddbec1de" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37271 } }
I20260810 18:05:41.300483 18952 raft_consensus.cc:374] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:41.300565 18952 raft_consensus.cc:729] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b231238800b24ccd82382fbfddbec1de, State: Initialized, Role: FOLLOWER
Aug 10 18:05:41 dist-test-slave-srgv krb5kdc[18867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:41.301090 18952 consensus_queue.cc:260] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b231238800b24ccd82382fbfddbec1de" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37271 } }
I20260810 18:05:41.301252 18952 raft_consensus.cc:388] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:41.301324 18952 raft_consensus.cc:482] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:41.301448 18952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:41.302749 18952 raft_consensus.cc:504] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b231238800b24ccd82382fbfddbec1de" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37271 } }
I20260810 18:05:41.303174 18952 leader_election.cc:302] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b231238800b24ccd82382fbfddbec1de; no voters:
I20260810 18:05:41.303553 18952 leader_election.cc:288] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:41.303674 18957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:41.303933 18957 raft_consensus.cc:686] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [term 1 LEADER]: Becoming Leader. State: Replica: b231238800b24ccd82382fbfddbec1de, State: Running, Role: LEADER
I20260810 18:05:41.304323 18957 consensus_queue.cc:237] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b231238800b24ccd82382fbfddbec1de" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37271 } }
I20260810 18:05:41.305037 18952 sys_catalog.cc:564] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:41.306173 18958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b231238800b24ccd82382fbfddbec1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b231238800b24ccd82382fbfddbec1de" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37271 } } }
I20260810 18:05:41.306314 18958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:41.306852 18966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:41.306843 18959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [sys.catalog]: SysCatalogTable state changed. Reason: New leader b231238800b24ccd82382fbfddbec1de. Latest consensus state: current_term: 1 leader_uuid: "b231238800b24ccd82382fbfddbec1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b231238800b24ccd82382fbfddbec1de" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 37271 } } }
I20260810 18:05:41.306983 18959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:41.307752 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:41.292527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40827 (local address 127.28.83.126:37271)
0810 18:05:41.293085 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:41.293104 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:41.293147 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:05:41.294128 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:41.294146 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:41.294522 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:41.294889 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:41.295644 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:41.297091 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:41.298016 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:41.298760 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:41.298957 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:41.301699 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:41.301706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:41.301723 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:41.301758 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:41.303981 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:41.304685 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:41.304690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:41.304696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:41.304835 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:41.305237 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:41.305240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:41.305242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:41.305800 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:41.306049 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:41.306424 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 18:05:41.306681 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":202,"threads_started":1}
I20260810 18:05:41.310452 18966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:41.310774 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:41.319092 18966 catalog_manager.cc:1347] Generated new cluster ID: fca1757f6d9440849876fb7fa29578c0
I20260810 18:05:41.319494 18966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:41.321830 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:41.313099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40977 (local address 127.28.83.126:37271)
0810 18:05:41.313197 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:41.313201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:41.313241 (+ 40us) server_negotiation.cc:407] Connection header received
0810 18:05:41.313290 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:41.313293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:41.313340 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:41.313449 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:41.314335 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:41.314959 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:41.316230 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:41.316468 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:41.316547 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:41.318281 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:41.318285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:41.318287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:41.318313 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:41.319991 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:41.320481 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:41.320489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:41.320491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:41.320564 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:41.320893 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:41.320897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:41.320898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:41.321068 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:41.321235 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:41.321551 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 18:05:41.321649 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 18:05:41.336841 18966 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:41.337963 18966 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:41.345757 18966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b231238800b24ccd82382fbfddbec1de: Generated new TSK 0
I20260810 18:05:41.346618 18966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:41.350219 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 18881
W20260810 18:05:41.356559 18865 connection.cc:535] client connection to 127.28.83.126:37271 recv error: Network error: failed to read from TLS socket (remote: 127.28.83.126:37271): Connection reset by peer (error 104)
2026-08-10T18:05:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:41 dist-test-slave-srgv krb5kdc[18983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:41 dist-test-slave-srgv krb5kdc[18983](info): set up 2 sockets
Aug 10 18:05:41 dist-test-slave-srgv krb5kdc[18983](info): commencing operation
krb5kdc: starting...
W20260810 18:05:42.412696 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:42 dist-test-slave-srgv krb5kdc[18983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:42.542722 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:46303
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:46303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:42.657184 18997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:42.657563 18997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:42.657657 18997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:42.661659 18997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:42.661760 18997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:42.661799 18997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:42.661836 18997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:42.661867 18997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:42.667269 18997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:46303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:46303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:42.668385 18997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:42.669442 18997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:42.677886 19003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:42.678030 18997 server_base.cc:1034] running on GCE node
W20260810 18:05:42.677928 19002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:42.677881 19005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:42.678833 18997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:42.679982 18997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:42.681178 18997 hybrid_clock.cc:648] HybridClock initialized: now 1786385142681150 us; error 47 us; skew 500 ppm
Aug 10 18:05:42 dist-test-slave-srgv krb5kdc[18983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:42.684082 18997 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:42.685098 18997 webserver.cc:460] Webserver started at http://127.28.83.126:44271/ using document root <none> and password file <none>
I20260810 18:05:42.685771 18997 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:42.685870 18997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:42.686117 18997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:42.688274 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "ed2580577e6841389f18e66f80d32d8b"
format_stamp: "Formatted at 2026-08-10 18:05:42 on dist-test-slave-srgv"
server_key: "54ea55a637933ca77535747ecb8bbc1f"
server_key_iv: "f2ebd343fd19ca6c38c0cd7232f035f5"
server_key_version: "encryptionkey@0"
I20260810 18:05:42.688905 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "ed2580577e6841389f18e66f80d32d8b"
format_stamp: "Formatted at 2026-08-10 18:05:42 on dist-test-slave-srgv"
server_key: "54ea55a637933ca77535747ecb8bbc1f"
server_key_iv: "f2ebd343fd19ca6c38c0cd7232f035f5"
server_key_version: "encryptionkey@0"
I20260810 18:05:42.692898 18997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 18:05:42.695554 19013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:42.696866 18997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:05:42.697017 18997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "ed2580577e6841389f18e66f80d32d8b"
format_stamp: "Formatted at 2026-08-10 18:05:42 on dist-test-slave-srgv"
server_key: "54ea55a637933ca77535747ecb8bbc1f"
server_key_iv: "f2ebd343fd19ca6c38c0cd7232f035f5"
server_key_version: "encryptionkey@0"
I20260810 18:05:42.697149 18997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:42.727972 18997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:42.731195 18997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:42.731420 18997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:42.738844 18997 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:46303
I20260810 18:05:42.738871 19068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:46303 every 8 connection(s)
I20260810 18:05:42.739868 18997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:42.740265 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 18997
I20260810 18:05:42.740432 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:42.740720 29005 external_mini_cluster.cc:1428] Setting key 7ec07f8c1db9168d5f1f5e54e1a19635
I20260810 18:05:42.743091 19069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:42.749092 19069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b: Bootstrap starting.
I20260810 18:05:42.749975 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:42.742193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37625 (local address 127.28.83.126:46303)
0810 18:05:42.742619 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:42.742639 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:42.742684 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:05:42.743382 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:42.743399 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:42.743737 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:42.744091 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:42.744491 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:42.745558 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:42.746487 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:42.746973 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:42.747153 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:42.748156 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:42.748161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:42.748173 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:42.748561 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:42.748627 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:42.748650 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:42.748795 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:42.748983 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 18:05:42.749127 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":149,"threads_started":1}
I20260810 18:05:42.751420 19069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:42.752287 19069 log.cc:824] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:42.752918 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:42.754374 19069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b: No bootstrap required, opened a new log
I20260810 18:05:42.758045 19069 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2580577e6841389f18e66f80d32d8b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46303 } }
I20260810 18:05:42.758358 19069 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:42.758446 19069 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2580577e6841389f18e66f80d32d8b, State: Initialized, Role: FOLLOWER
I20260810 18:05:42.758924 19069 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2580577e6841389f18e66f80d32d8b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46303 } }
I20260810 18:05:42.759081 19069 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:42.759161 19069 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:42.759264 19069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:42.759931 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:42.755097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53073 (local address 127.28.83.126:46303)
0810 18:05:42.755220 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:42.755224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:42.755392 (+ 168us) server_negotiation.cc:407] Connection header received
0810 18:05:42.755509 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:42.755511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:42.755551 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:42.755628 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:42.755915 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:42.756989 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:42.758137 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:42.758390 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:42.758511 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:42.759428 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:42.759432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:42.759434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:42.759452 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:42.759475 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:42.759485 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:42.759553 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:42.759739 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 18:05:42.759771 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:05:42.760550 19069 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2580577e6841389f18e66f80d32d8b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46303 } }
I20260810 18:05:42.760957 19069 leader_election.cc:302] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed2580577e6841389f18e66f80d32d8b; no voters:
I20260810 18:05:42.761349 19069 leader_election.cc:288] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:42.761482 19079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:42.761701 19079 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [term 1 LEADER]: Becoming Leader. State: Replica: ed2580577e6841389f18e66f80d32d8b, State: Running, Role: LEADER
I20260810 18:05:42.762050 19079 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2580577e6841389f18e66f80d32d8b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46303 } }
I20260810 18:05:42.762669 19069 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:42.763907 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2580577e6841389f18e66f80d32d8b. Latest consensus state: current_term: 1 leader_uuid: "ed2580577e6841389f18e66f80d32d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2580577e6841389f18e66f80d32d8b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46303 } } }
I20260810 18:05:42.764031 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:42.764322 19080 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed2580577e6841389f18e66f80d32d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2580577e6841389f18e66f80d32d8b" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46303 } } }
I20260810 18:05:42.764410 19080 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:42.764497 19088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:42.767457 19088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:42.773943 19088 catalog_manager.cc:1347] Generated new cluster ID: 259c061a7249403ba8578298ddef6574
I20260810 18:05:42.774032 19088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:42.778559 19088 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:42.779781 19088 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:42.786491 19088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2580577e6841389f18e66f80d32d8b: Generated new TSK 0
I20260810 18:05:42.787384 19088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:42.791462 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 18997
W20260810 18:05:42.798095 18981 connection.cc:535] client connection to 127.28.83.126:46303 recv error: Network error: failed to read from TLS socket (remote: 127.28.83.126:46303): Connection reset by peer (error 104)
2026-08-10T18:05:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:42 dist-test-slave-srgv krb5kdc[19101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:42 dist-test-slave-srgv krb5kdc[19101](info): set up 2 sockets
Aug 10 18:05:42 dist-test-slave-srgv krb5kdc[19101](info): commencing operation
krb5kdc: starting...
W20260810 18:05:43.844826 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:43 dist-test-slave-srgv krb5kdc[19101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:43.977167 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43541
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:43541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:44.098248 19115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:44.098567 19115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:44.098613 19115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:44.102811 19115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:44.102888 19115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:44.102908 19115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:44.102926 19115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:44.102979 19115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:44.108680 19115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:43541
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:44.109938 19115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:44.111001 19115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:44.119840 19121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:44.119844 19123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:44.119840 19120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:44.120103 19115 server_base.cc:1034] running on GCE node
I20260810 18:05:44.120842 19115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:44.122083 19115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:44.123289 19115 hybrid_clock.cc:648] HybridClock initialized: now 1786385144123265 us; error 44 us; skew 500 ppm
Aug 10 18:05:44 dist-test-slave-srgv krb5kdc[19101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385144, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:44.126247 19115 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:44.127226 19115 webserver.cc:460] Webserver started at http://127.28.83.126:40839/ using document root <none> and password file <none>
I20260810 18:05:44.127813 19115 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:44.127908 19115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:44.128147 19115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:44.130381 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "ba3beddc36534e8787a546fee7e7c2ec"
format_stamp: "Formatted at 2026-08-10 18:05:44 on dist-test-slave-srgv"
server_key: "55988ed9c2ae119e2f83e1ded1a107f6"
server_key_iv: "170be53a31113778d4a777fd9a1c0f05"
server_key_version: "encryptionkey@0"
I20260810 18:05:44.131095 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "ba3beddc36534e8787a546fee7e7c2ec"
format_stamp: "Formatted at 2026-08-10 18:05:44 on dist-test-slave-srgv"
server_key: "55988ed9c2ae119e2f83e1ded1a107f6"
server_key_iv: "170be53a31113778d4a777fd9a1c0f05"
server_key_version: "encryptionkey@0"
I20260810 18:05:44.134975 19115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:05:44.137507 19130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:44.138733 19115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:05:44.138885 19115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "ba3beddc36534e8787a546fee7e7c2ec"
format_stamp: "Formatted at 2026-08-10 18:05:44 on dist-test-slave-srgv"
server_key: "55988ed9c2ae119e2f83e1ded1a107f6"
server_key_iv: "170be53a31113778d4a777fd9a1c0f05"
server_key_version: "encryptionkey@0"
I20260810 18:05:44.139003 19115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:44.143698 19115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:44.147012 19115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:44.147217 19115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:44.154659 19115 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:43541
I20260810 18:05:44.154690 19182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:43541 every 8 connection(s)
I20260810 18:05:44.155699 19115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:44.158666 19183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:44.164369 19183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec: Bootstrap starting.
I20260810 18:05:44.164628 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 19115
I20260810 18:05:44.164772 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:44.165057 29005 external_mini_cluster.cc:1428] Setting key 7fb2a4f3e8843bb405a9cbf4fb8b2ddc
I20260810 18:05:44.167358 19183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:44.168303 19183 log.cc:824] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:44.170704 19183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec: No bootstrap required, opened a new log
I20260810 18:05:44.171101 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:44.166494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40331 (local address 127.28.83.126:43541)
0810 18:05:44.166976 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:44.166995 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:44.167049 (+ 54us) server_negotiation.cc:407] Connection header received
0810 18:05:44.167971 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:44.168007 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:44.168409 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:44.168804 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:44.169721 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:44.169726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:44.169738 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:44.169748 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:44.169803 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:44.169821 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:44.169913 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:44.170160 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 18:05:44.170341 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":133,"threads_started":1}
I20260810 18:05:44.174032 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:44.174962 19183 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3beddc36534e8787a546fee7e7c2ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43541 } }
I20260810 18:05:44.175218 19183 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:44.175274 19183 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3beddc36534e8787a546fee7e7c2ec, State: Initialized, Role: FOLLOWER
I20260810 18:05:44.175745 19183 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3beddc36534e8787a546fee7e7c2ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43541 } }
I20260810 18:05:44.175906 19183 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:44.175984 19183 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:44.176111 19183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:44.177552 19183 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3beddc36534e8787a546fee7e7c2ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43541 } }
I20260810 18:05:44.178013 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:44.176214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58343 (local address 127.28.83.126:43541)
0810 18:05:44.176341 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:44.176346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:44.176576 (+ 230us) server_negotiation.cc:407] Connection header received
0810 18:05:44.176696 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:44.176698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:44.176732 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:44.176800 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:44.177596 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:44.177599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:44.177601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:44.177612 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:44.177639 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:44.177650 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:44.177770 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:44.177846 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 18:05:44.177881 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 18:05:44.177974 19183 leader_election.cc:302] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba3beddc36534e8787a546fee7e7c2ec; no voters:
I20260810 18:05:44.178406 19183 leader_election.cc:288] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:44.178529 19193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:44.178735 19193 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [term 1 LEADER]: Becoming Leader. State: Replica: ba3beddc36534e8787a546fee7e7c2ec, State: Running, Role: LEADER
I20260810 18:05:44.179060 19193 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3beddc36534e8787a546fee7e7c2ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43541 } }
I20260810 18:05:44.179721 19183 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:44.180254 19193 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba3beddc36534e8787a546fee7e7c2ec. Latest consensus state: current_term: 1 leader_uuid: "ba3beddc36534e8787a546fee7e7c2ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3beddc36534e8787a546fee7e7c2ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43541 } } }
I20260810 18:05:44.180369 19193 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:44.180739 19198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:44.180879 19194 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba3beddc36534e8787a546fee7e7c2ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3beddc36534e8787a546fee7e7c2ec" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43541 } } }
I20260810 18:05:44.181043 19194 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:44.183859 19198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:44.189875 19198 catalog_manager.cc:1347] Generated new cluster ID: b03f43e213644655b219ca6fdc78019b
I20260810 18:05:44.189970 19198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:44.224431 19198 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:44.224790 19198 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:44.235565 19198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba3beddc36534e8787a546fee7e7c2ec: Generated new TSK 0
I20260810 18:05:44.236318 19198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:44.247865 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 19115
2026-08-10T18:05:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:44 dist-test-slave-srgv krb5kdc[19216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:44 dist-test-slave-srgv krb5kdc[19216](info): set up 2 sockets
Aug 10 18:05:44 dist-test-slave-srgv krb5kdc[19216](info): commencing operation
krb5kdc: starting...
W20260810 18:05:45.309011 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:45 dist-test-slave-srgv krb5kdc[19216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:45.440781 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40593
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:45.568907 19231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:45.569296 19231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:45.569458 19231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:45.574005 19231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:45.574090 19231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:45.574110 19231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:45.574127 19231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:45.574142 19231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:45.579931 19231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:40593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:45.581056 19231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:45.582159 19231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:45.590950 19237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:45.591096 19231 server_base.cc:1034] running on GCE node
W20260810 18:05:45.590910 19239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:45.590924 19236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:45.592031 19231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:45.593149 19231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:45.594321 19231 hybrid_clock.cc:648] HybridClock initialized: now 1786385145594305 us; error 32 us; skew 500 ppm
Aug 10 18:05:45 dist-test-slave-srgv krb5kdc[19216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385145, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:45.597172 19231 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:45.598172 19231 webserver.cc:460] Webserver started at http://127.28.83.126:42903/ using document root <none> and password file <none>
I20260810 18:05:45.598723 19231 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:45.598796 19231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:45.598975 19231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:45.601104 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "ea185b15a5424ae9b15ea12c93ddadcf"
format_stamp: "Formatted at 2026-08-10 18:05:45 on dist-test-slave-srgv"
server_key: "da46b6ba7f2f17b8ccc74bc0e72ec43b"
server_key_iv: "cb45f36413e0787896747e5a885b2456"
server_key_version: "encryptionkey@0"
I20260810 18:05:45.601752 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "ea185b15a5424ae9b15ea12c93ddadcf"
format_stamp: "Formatted at 2026-08-10 18:05:45 on dist-test-slave-srgv"
server_key: "da46b6ba7f2f17b8ccc74bc0e72ec43b"
server_key_iv: "cb45f36413e0787896747e5a885b2456"
server_key_version: "encryptionkey@0"
I20260810 18:05:45.605777 19231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 18:05:45.608314 19247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:45.609684 19231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 18:05:45.609848 19231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "ea185b15a5424ae9b15ea12c93ddadcf"
format_stamp: "Formatted at 2026-08-10 18:05:45 on dist-test-slave-srgv"
server_key: "da46b6ba7f2f17b8ccc74bc0e72ec43b"
server_key_iv: "cb45f36413e0787896747e5a885b2456"
server_key_version: "encryptionkey@0"
I20260810 18:05:45.609980 19231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:45.620234 19231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:45.623518 19231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:45.623764 19231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:45.631965 19231 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:40593
I20260810 18:05:45.631974 19301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:40593 every 8 connection(s)
I20260810 18:05:45.632997 19231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:45.635938 19302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:45.638378 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 19231
I20260810 18:05:45.638536 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:45.638842 29005 external_mini_cluster.cc:1428] Setting key f06c9c9055053d92e6ed61eacd04ee11
I20260810 18:05:45.642769 19302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf: Bootstrap starting.
I20260810 18:05:45.645522 19302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:45.646410 19302 log.cc:824] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:05:45 dist-test-slave-srgv krb5kdc[19216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:45.648942 19302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf: No bootstrap required, opened a new log
I20260810 18:05:45.653560 19302 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea185b15a5424ae9b15ea12c93ddadcf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40593 } }
I20260810 18:05:45.653873 19302 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:45.653978 19302 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea185b15a5424ae9b15ea12c93ddadcf, State: Initialized, Role: FOLLOWER
I20260810 18:05:45.654479 19302 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea185b15a5424ae9b15ea12c93ddadcf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40593 } }
I20260810 18:05:45.654654 19302 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:45.654729 19302 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:45.654827 19302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:45.655198 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:45.640306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43537 (local address 127.28.83.126:40593)
0810 18:05:45.640740 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:45.640757 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:45.640794 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:05:45.641688 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:45.641709 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:45.642072 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:45.642373 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:45.643363 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:45.644384 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:45.645616 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:45.646121 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:45.646262 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:45.649206 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:45.649213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:45.649231 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:45.649264 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:45.651892 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:45.652353 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:45.652359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:45.652365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:45.652512 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:45.652779 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:45.652782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:45.652784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:45.653338 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:45.653593 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:45.653797 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 18:05:45.654126 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
I20260810 18:05:45.655979 19302 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea185b15a5424ae9b15ea12c93ddadcf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40593 } }
I20260810 18:05:45.656330 19302 leader_election.cc:302] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea185b15a5424ae9b15ea12c93ddadcf; no voters:
I20260810 18:05:45.656653 19302 leader_election.cc:288] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:45.656785 19308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:45.657020 19308 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [term 1 LEADER]: Becoming Leader. State: Replica: ea185b15a5424ae9b15ea12c93ddadcf, State: Running, Role: LEADER
I20260810 18:05:45.657457 19308 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea185b15a5424ae9b15ea12c93ddadcf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40593 } }
I20260810 18:05:45.658114 19302 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:45.658406 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:45.659341 19309 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea185b15a5424ae9b15ea12c93ddadcf. Latest consensus state: current_term: 1 leader_uuid: "ea185b15a5424ae9b15ea12c93ddadcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea185b15a5424ae9b15ea12c93ddadcf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40593 } } }
I20260810 18:05:45.659436 19309 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:45.659706 19312 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea185b15a5424ae9b15ea12c93ddadcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea185b15a5424ae9b15ea12c93ddadcf" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 40593 } } }
I20260810 18:05:45.659783 19312 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:45.659827 19319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:45.662895 19319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:45.670192 19319 catalog_manager.cc:1347] Generated new cluster ID: 34011003d9a749bbb51b5a628bd0f24e
I20260810 18:05:45.670284 19319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:45.675091 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:45.660849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57481 (local address 127.28.83.126:40593)
0810 18:05:45.661127 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:45.661132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:45.665810 (+ 4678us) server_negotiation.cc:407] Connection header received
0810 18:05:45.665970 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:45.665975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:45.666032 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:45.666134 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:45.667842 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:45.668536 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:45.669767 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:45.670112 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:45.670219 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:45.671809 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:45.671813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:45.671816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:45.671849 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:45.673339 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:45.673838 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:45.673841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:45.673843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:45.673927 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:45.674208 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:45.674211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:45.674213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:45.674392 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:45.674531 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:45.674786 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 18:05:45.674906 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260810 18:05:45.696691 19319 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:45.698019 19319 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:45.702796 19319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea185b15a5424ae9b15ea12c93ddadcf: Generated new TSK 0
I20260810 18:05:45.703487 19319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:45.709033 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 19231
2026-08-10T18:05:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:45 dist-test-slave-srgv krb5kdc[19336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:45 dist-test-slave-srgv krb5kdc[19336](info): set up 2 sockets
Aug 10 18:05:45 dist-test-slave-srgv krb5kdc[19336](info): commencing operation
krb5kdc: starting...
W20260810 18:05:46.770226 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:46 dist-test-slave-srgv krb5kdc[19336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:46.898408 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:45859
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:47.015336 19350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:47.015625 19350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:47.015681 19350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:47.019847 19350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:47.019925 19350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:47.019946 19350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:47.019992 19350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:47.020032 19350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:47.025671 19350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:47.026863 19350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:47.027937 19350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:47.036975 19359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:47.037185 19350 server_base.cc:1034] running on GCE node
W20260810 18:05:47.036968 19357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:47.036968 19356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:47.038069 19350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:47.039218 19350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:47.040386 19350 hybrid_clock.cc:648] HybridClock initialized: now 1786385147040361 us; error 42 us; skew 500 ppm
Aug 10 18:05:47 dist-test-slave-srgv krb5kdc[19336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:47.043504 19350 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:47.044476 19350 webserver.cc:460] Webserver started at http://127.28.83.126:44549/ using document root <none> and password file <none>
I20260810 18:05:47.045118 19350 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:47.045188 19350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:47.045485 19350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:47.047629 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "52c5e1745d9b404385d8a63c0698bee5"
format_stamp: "Formatted at 2026-08-10 18:05:47 on dist-test-slave-srgv"
server_key: "81043a3600d6b62867ce3f7d79fef3a4"
server_key_iv: "6cc5606dadacfe5dca782632bd787d4f"
server_key_version: "encryptionkey@0"
I20260810 18:05:47.048247 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "52c5e1745d9b404385d8a63c0698bee5"
format_stamp: "Formatted at 2026-08-10 18:05:47 on dist-test-slave-srgv"
server_key: "81043a3600d6b62867ce3f7d79fef3a4"
server_key_iv: "6cc5606dadacfe5dca782632bd787d4f"
server_key_version: "encryptionkey@0"
I20260810 18:05:47.052408 19350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 18:05:47.054943 19366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:47.056236 19350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:05:47.056389 19350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "52c5e1745d9b404385d8a63c0698bee5"
format_stamp: "Formatted at 2026-08-10 18:05:47 on dist-test-slave-srgv"
server_key: "81043a3600d6b62867ce3f7d79fef3a4"
server_key_iv: "6cc5606dadacfe5dca782632bd787d4f"
server_key_version: "encryptionkey@0"
I20260810 18:05:47.056512 19350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:47.088248 19350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:47.091928 19350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:47.092164 19350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:47.099836 19350 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:45859
I20260810 18:05:47.099865 19421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:45859 every 8 connection(s)
I20260810 18:05:47.100840 19350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:47.103646 19422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:47.106335 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 19350
I20260810 18:05:47.106484 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:47.106823 29005 external_mini_cluster.cc:1428] Setting key ab2e101c2afc9c024de4155753d4d98e
I20260810 18:05:47.110122 19422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5: Bootstrap starting.
I20260810 18:05:47.112759 19422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:47.113660 19422 log.cc:824] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:47.115990 19422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5: No bootstrap required, opened a new log
I20260810 18:05:47.116428 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:47.108344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44349 (local address 127.28.83.126:45859)
0810 18:05:47.108833 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:47.108848 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:47.108889 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:05:47.109619 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:47.109641 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:47.110001 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:47.110351 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:47.110695 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:47.111675 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:47.112945 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:47.113510 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:47.113653 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:47.114594 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:47.114602 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:47.114634 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:47.115045 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:47.115103 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:47.115129 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:47.115208 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:47.115384 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 18:05:47.115629 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":143,"threads_started":1}
I20260810 18:05:47.119503 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:47.120361 19422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c5e1745d9b404385d8a63c0698bee5" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45859 } }
I20260810 18:05:47.120678 19422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:47.120741 19422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5e1745d9b404385d8a63c0698bee5, State: Initialized, Role: FOLLOWER
I20260810 18:05:47.121286 19422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c5e1745d9b404385d8a63c0698bee5" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45859 } }
I20260810 18:05:47.121471 19422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:47.121546 19422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:47.121685 19422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:47.122982 19422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c5e1745d9b404385d8a63c0698bee5" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45859 } }
I20260810 18:05:47.123338 19422 leader_election.cc:302] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c5e1745d9b404385d8a63c0698bee5; no voters:
I20260810 18:05:47.123682 19422 leader_election.cc:288] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:47.123802 19432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:47.124025 19432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [term 1 LEADER]: Becoming Leader. State: Replica: 52c5e1745d9b404385d8a63c0698bee5, State: Running, Role: LEADER
I20260810 18:05:47.124416 19432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c5e1745d9b404385d8a63c0698bee5" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45859 } }
I20260810 18:05:47.125064 19422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:47.125638 19432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c5e1745d9b404385d8a63c0698bee5. Latest consensus state: current_term: 1 leader_uuid: "52c5e1745d9b404385d8a63c0698bee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c5e1745d9b404385d8a63c0698bee5" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45859 } } }
I20260810 18:05:47.125789 19432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:47.126291 19437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:47.126844 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:47.121957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34017 (local address 127.28.83.126:45859)
0810 18:05:47.122100 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:47.122104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:47.122280 (+ 176us) server_negotiation.cc:407] Connection header received
0810 18:05:47.122364 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:47.122367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:47.122413 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:47.122502 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:47.122745 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:47.123536 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:47.124672 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:47.124979 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:47.125100 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:47.126014 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:47.126019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:47.126022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:47.126038 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:47.126072 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:47.126101 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:47.126184 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:47.126620 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 18:05:47.126673 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:05:47.127732 19434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52c5e1745d9b404385d8a63c0698bee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c5e1745d9b404385d8a63c0698bee5" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 45859 } } }
I20260810 18:05:47.127835 19434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:47.130100 19437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:47.136125 19437 catalog_manager.cc:1347] Generated new cluster ID: 88f7238d5c754aed8d8219c19c1e77bd
I20260810 18:05:47.136201 19437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:47.149092 19437 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:47.150219 19437 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:47.160856 19437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52c5e1745d9b404385d8a63c0698bee5: Generated new TSK 0
I20260810 18:05:47.161712 19437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:47.173846 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 19350
2026-08-10T18:05:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:47 dist-test-slave-srgv krb5kdc[19454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:47 dist-test-slave-srgv krb5kdc[19454](info): set up 2 sockets
Aug 10 18:05:47 dist-test-slave-srgv krb5kdc[19454](info): commencing operation
krb5kdc: starting...
W20260810 18:05:48.240985 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:48 dist-test-slave-srgv krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:48.377285 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:46007
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:46007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:48.491001 19467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:48.491338 19467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:48.491425 19467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:48.495500 19467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:48.495609 19467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:48.495647 19467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:48.495687 19467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:48.495723 19467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:48.501336 19467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:46007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:46007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:48.502534 19467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:48.503595 19467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:48.512504 19473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:48.512508 19472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:48.512507 19475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:48.512717 19467 server_base.cc:1034] running on GCE node
I20260810 18:05:48.513590 19467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:48.514777 19467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:48.516003 19467 hybrid_clock.cc:648] HybridClock initialized: now 1786385148515976 us; error 44 us; skew 500 ppm
Aug 10 18:05:48 dist-test-slave-srgv krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:48.518982 19467 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:48.519886 19467 webserver.cc:460] Webserver started at http://127.28.83.126:36655/ using document root <none> and password file <none>
I20260810 18:05:48.520432 19467 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:48.520493 19467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:48.520660 19467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:48.522850 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "6401a871b931467da883a1ab1ed68bbc"
format_stamp: "Formatted at 2026-08-10 18:05:48 on dist-test-slave-srgv"
server_key: "6df321a26680d79bb889691496f936ed"
server_key_iv: "77538713903583f05aa38e5fb1155d44"
server_key_version: "encryptionkey@0"
I20260810 18:05:48.523432 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "6401a871b931467da883a1ab1ed68bbc"
format_stamp: "Formatted at 2026-08-10 18:05:48 on dist-test-slave-srgv"
server_key: "6df321a26680d79bb889691496f936ed"
server_key_iv: "77538713903583f05aa38e5fb1155d44"
server_key_version: "encryptionkey@0"
I20260810 18:05:48.527364 19467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:05:48.529888 19482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:48.531183 19467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:05:48.531308 19467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "6401a871b931467da883a1ab1ed68bbc"
format_stamp: "Formatted at 2026-08-10 18:05:48 on dist-test-slave-srgv"
server_key: "6df321a26680d79bb889691496f936ed"
server_key_iv: "77538713903583f05aa38e5fb1155d44"
server_key_version: "encryptionkey@0"
I20260810 18:05:48.531406 19467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:48.536226 19467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:48.539323 19467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:48.539489 19467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:48.546778 19467 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:46007
I20260810 18:05:48.546873 19535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:46007 every 8 connection(s)
I20260810 18:05:48.547788 19467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:48.550607 19536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:48.554687 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 19467
I20260810 18:05:48.554885 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:48.555205 29005 external_mini_cluster.cc:1428] Setting key 47d90b884caafdb192a3433ebcd31cc7
I20260810 18:05:48.556653 19536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc: Bootstrap starting.
I20260810 18:05:48.559195 19536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:48.560139 19536 log.cc:824] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:48.560818 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:48.556624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50867 (local address 127.28.83.126:46007)
0810 18:05:48.557123 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:48.557143 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:48.557181 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:05:48.557923 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:48.557941 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:48.558242 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:48.558557 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:48.559482 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:48.559487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:48.559499 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:48.559511 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:48.559572 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:48.559592 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:48.559682 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:48.559815 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 18:05:48.559986 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":150,"threads_started":1}
I20260810 18:05:48.562270 19536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc: No bootstrap required, opened a new log
I20260810 18:05:48.564028 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:48.567274 19536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401a871b931467da883a1ab1ed68bbc" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46007 } }
I20260810 18:05:48.567615 19536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:05:48.567699 19536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6401a871b931467da883a1ab1ed68bbc, State: Initialized, Role: FOLLOWER
I20260810 18:05:48.568254 19536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401a871b931467da883a1ab1ed68bbc" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46007 } }
I20260810 18:05:48.568472 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:48.566598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37003 (local address 127.28.83.126:46007)
0810 18:05:48.566721 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:48.566725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:48.566965 (+ 240us) server_negotiation.cc:407] Connection header received
0810 18:05:48.567091 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:48.567093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:48.567127 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:48.567209 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:48.568067 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:48.568070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:48.568072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:05:48.568082 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:48.568108 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:05:48.568119 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:48.568191 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:48.568307 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 18:05:48.568339 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 18:05:48.568547 19536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:05:48.568638 19536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:05:48.568753 19536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:48.570016 19536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401a871b931467da883a1ab1ed68bbc" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46007 } }
I20260810 18:05:48.570401 19536 leader_election.cc:302] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6401a871b931467da883a1ab1ed68bbc; no voters:
I20260810 18:05:48.570726 19536 leader_election.cc:288] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:05:48.570868 19547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:48.571116 19547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [term 1 LEADER]: Becoming Leader. State: Replica: 6401a871b931467da883a1ab1ed68bbc, State: Running, Role: LEADER
I20260810 18:05:48.571550 19547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401a871b931467da883a1ab1ed68bbc" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46007 } }
I20260810 18:05:48.572057 19536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:48.573369 19549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6401a871b931467da883a1ab1ed68bbc. Latest consensus state: current_term: 1 leader_uuid: "6401a871b931467da883a1ab1ed68bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401a871b931467da883a1ab1ed68bbc" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46007 } } }
I20260810 18:05:48.573526 19549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:48.573858 19548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6401a871b931467da883a1ab1ed68bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401a871b931467da883a1ab1ed68bbc" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 46007 } } }
I20260810 18:05:48.573940 19548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:48.574313 19556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:48.577580 19556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:48.583067 19556 catalog_manager.cc:1347] Generated new cluster ID: 6d72224254434346912e6247b0051b8c
I20260810 18:05:48.583153 19556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:48.593108 19556 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:48.593523 19556 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:48.599767 19556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6401a871b931467da883a1ab1ed68bbc: Generated new TSK 0
I20260810 18:05:48.600513 19556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:05:48.607115 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 19467
2026-08-10T18:05:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1432 ms)
[----------] 18 tests from AuthTokenIssuingTest (26119 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:48 dist-test-slave-srgv krb5kdc[19570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:48 dist-test-slave-srgv krb5kdc[19570](info): set up 2 sockets
Aug 10 18:05:48 dist-test-slave-srgv krb5kdc[19570](info): commencing operation
krb5kdc: starting...
W20260810 18:05:49.660669 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:49 dist-test-slave-srgv krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:49.792126 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33543
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:33543,127.28.83.125:42715,127.28.83.124:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:49.907964 19585 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:05:49.908299 19585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:49.908325 19585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:49.908435 19585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:49.912647 19585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:49.912739 19585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:49.912760 19585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:49.912777 19585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:49.912792 19585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:49.918443 19585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:33543,127.28.83.125:42715,127.28.83.124:38729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:33543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:49.919657 19585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:49.920773 19585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:49.929839 19590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:49.929831 19591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:49.929831 19593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:49.930194 19585 server_base.cc:1034] running on GCE node
I20260810 18:05:49.930985 19585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:49.932251 19585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:49.933482 19585 hybrid_clock.cc:648] HybridClock initialized: now 1786385149933475 us; error 34 us; skew 500 ppm
Aug 10 18:05:49 dist-test-slave-srgv krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:49.936479 19585 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:49.937554 19585 webserver.cc:460] Webserver started at http://127.28.83.126:43805/ using document root <none> and password file <none>
I20260810 18:05:49.938165 19585 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:49.938266 19585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:49.938519 19585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:49.940827 19585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "88166728902a4ef6a672ca8074722ee9"
format_stamp: "Formatted at 2026-08-10 18:05:49 on dist-test-slave-srgv"
server_key: "2a11fc39d650958ecba528531dcd079f"
server_key_iv: "de6b529733239be4ed7d07d54481ca24"
server_key_version: "encryptionkey@0"
I20260810 18:05:49.941509 19585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "88166728902a4ef6a672ca8074722ee9"
format_stamp: "Formatted at 2026-08-10 18:05:49 on dist-test-slave-srgv"
server_key: "2a11fc39d650958ecba528531dcd079f"
server_key_iv: "de6b529733239be4ed7d07d54481ca24"
server_key_version: "encryptionkey@0"
I20260810 18:05:49.946208 19585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:05:49.949999 19601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:49.952144 19585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:05:49.952375 19585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "88166728902a4ef6a672ca8074722ee9"
format_stamp: "Formatted at 2026-08-10 18:05:49 on dist-test-slave-srgv"
server_key: "2a11fc39d650958ecba528531dcd079f"
server_key_iv: "de6b529733239be4ed7d07d54481ca24"
server_key_version: "encryptionkey@0"
I20260810 18:05:49.952618 19585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:49.971875 19585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:49.975679 19585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:49.975921 19585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:49.983582 19585 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:33543
I20260810 18:05:49.983924 19653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:33543 every 8 connection(s)
I20260810 18:05:49.984761 19585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:49.987871 19654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:49.990373 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 19585
I20260810 18:05:49.990538 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:49.990856 29005 external_mini_cluster.cc:1428] Setting key 003bd613fc7abfa4e18f027937e72db5
I20260810 18:05:49.991967 19654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.83.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.125@KRBTEST.COM" created.
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:50.010233 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:49.994334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33543 (local address 127.0.0.1:55708)
0810 18:05:49.994861 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:49.994917 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:49.995925 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:49.996795 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:49.996800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:49.997172 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:49.997543 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:49.997558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:49.999383 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:49.999386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.000255 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.000265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.001130 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.001133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.001217 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.001955 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:50.001985 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:50.003921 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:50.006700 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.006710 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.006719 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.006974 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.007426 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.007429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.007430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.007561 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.008478 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:50.008485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:50.008638 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 18:05:50.008831 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 18:05:50.009193 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":165,"threads_started":1}
I20260810 18:05:50.010233 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:49.995293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.28.83.126:33543)
0810 18:05:49.995652 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:49.995674 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:49.995723 (+ 49us) server_negotiation.cc:407] Connection header received
0810 18:05:49.996274 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:49.996283 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:49.996556 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:49.996797 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:49.997716 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:49.999224 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.000405 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.000955 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.001142 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.004135 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.004141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.004153 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.004175 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.006545 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.007109 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.007119 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.007123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.007284 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.007671 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.007677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.007682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.008314 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.008471 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.008956 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.009345 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":228,"thread_start_us":111,"threads_started":1}
I20260810 18:05:50.013207 19654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:50.014107 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.013784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42715 (local address 127.0.0.1:44036)
0810 18:05:50.013938 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.014023 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 18:05:50.014463 19603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
I20260810 18:05:50.015512 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.015236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42715 (local address 127.0.0.1:44050)
0810 18:05:50.015386 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.015449 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:05:50.015717 19654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.125:42715: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab.
Entry for principal kudu/127.28.83.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab.
I20260810 18:05:50.020704 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.125:42715
--webserver_interface=127.28.83.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab
--principal=kudu/127.28.83.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:33543,127.28.83.125:42715,127.28.83.124:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:05:50.065030 19654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } attempt: 1
I20260810 18:05:50.066149 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.065771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42715 (local address 127.0.0.1:44066)
0810 18:05:50.065979 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.066061 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 18:05:50.066905 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.066685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42715 (local address 127.0.0.1:44074)
0810 18:05:50.066791 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.066847 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 18:05:50.067037 19654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.125:42715: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
I20260810 18:05:50.135224 19654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } attempt: 2
I20260810 18:05:50.136118 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.135806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42715 (local address 127.0.0.1:44080)
0810 18:05:50.135962 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.136041 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 18:05:50.136837 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.136627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42715 (local address 127.0.0.1:44084)
0810 18:05:50.136729 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.136781 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 18:05:50.136969 19654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.125:42715: Network error: Client connection negotiation failed: client connection to 127.28.83.125:42715: connect: Connection refused (error 111)
W20260810 18:05:50.139437 19661 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:05:50.139825 19661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:50.139889 19661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:50.139971 19661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:50.144215 19661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:50.144323 19661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:50.144366 19661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:50.144403 19661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:50.144434 19661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:50.150218 19661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:33543,127.28.83.125:42715,127.28.83.124:38729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab
--principal=kudu/127.28.83.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.125:42715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.83.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:50.151432 19661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:50.152515 19661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:50.161515 19666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:50.161566 19669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:50.161546 19667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:50.161842 19661 server_base.cc:1034] running on GCE node
I20260810 18:05:50.162452 19661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:50.163632 19661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:50.164844 19661 hybrid_clock.cc:648] HybridClock initialized: now 1786385150164826 us; error 34 us; skew 500 ppm
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385150, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:50.168083 19661 init.cc:377] Logged in from keytab as kudu/127.28.83.125@KRBTEST.COM (short username kudu)
I20260810 18:05:50.169205 19661 webserver.cc:460] Webserver started at http://127.28.83.125:41037/ using document root <none> and password file <none>
I20260810 18:05:50.169895 19661 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:50.169996 19661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:50.170261 19661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:50.172539 19661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/data/instance:
uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa"
format_stamp: "Formatted at 2026-08-10 18:05:50 on dist-test-slave-srgv"
server_key: "f310b9bbc7132e7af2d358d3a155767f"
server_key_iv: "570b505e414564b7aa784a910ca2d4b1"
server_key_version: "encryptionkey@0"
I20260810 18:05:50.173192 19661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/wal/instance:
uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa"
format_stamp: "Formatted at 2026-08-10 18:05:50 on dist-test-slave-srgv"
server_key: "f310b9bbc7132e7af2d358d3a155767f"
server_key_iv: "570b505e414564b7aa784a910ca2d4b1"
server_key_version: "encryptionkey@0"
I20260810 18:05:50.177812 19661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:05:50.180583 19678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:50.181998 19661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:50.182194 19661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/wal
uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa"
format_stamp: "Formatted at 2026-08-10 18:05:50 on dist-test-slave-srgv"
server_key: "f310b9bbc7132e7af2d358d3a155767f"
server_key_iv: "570b505e414564b7aa784a910ca2d4b1"
server_key_version: "encryptionkey@0"
I20260810 18:05:50.182331 19661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:50.204078 19661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:50.207976 19661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:50.208227 19661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:50.216095 19661 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.125:42715
I20260810 18:05:50.216408 19731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.125:42715 every 8 connection(s)
I20260810 18:05:50.217262 19661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
I20260810 18:05:50.218976 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 19661
I20260810 18:05:50.219085 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-1/wal/instance
I20260810 18:05:50.219452 29005 external_mini_cluster.cc:1428] Setting key d93a9391ed390450d8f972f98b7f5c55
I20260810 18:05:50.220782 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:50.224964 19732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.83.124@KRBTEST.COM; defaulting to no policy
I20260810 18:05:50.228153 19654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.124@KRBTEST.COM" created.
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385150, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:50.241039 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.228633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42715 (local address 127.0.0.1:44096)
0810 18:05:50.228970 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.228985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:50.229075 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:50.229768 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:50.229771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:50.229977 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:50.230100 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.230108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.231928 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.231934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.232725 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.232741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.233458 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.233462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.233570 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.234150 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:50.234163 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:50.235733 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:50.238538 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.238544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.238548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.238792 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.239466 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.239469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.239471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.239627 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.240436 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:50.240439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:50.240553 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 18:05:50.240707 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 18:05:50.240863 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260810 18:05:50.241773 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.227104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.28.83.126:33543)
0810 18:05:50.227283 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.227288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.228251 (+ 963us) server_negotiation.cc:407] Connection header received
0810 18:05:50.229548 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.229551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.229654 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.229807 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.230810 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.231558 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.232975 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.233364 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.233540 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.236670 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.236676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.236683 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.236708 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.238297 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.238915 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.238919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.238921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.239021 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.239454 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.239457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.239459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.239770 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.239883 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.241488 (+ 1605us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.241615 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"spinlock_wait_cycles":896}
I20260810 18:05:50.242183 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.227539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33543 (local address 127.0.0.1:55718)
0810 18:05:50.228056 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.228120 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:50.229294 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:50.229870 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:50.229876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:50.230380 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:50.230647 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.230664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.231694 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.231701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.232850 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.232862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.233768 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.233775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.233939 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.234526 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:50.234552 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:50.236259 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:50.238433 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.238444 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.238457 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.238796 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.239159 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.239161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.239163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.239320 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.239885 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:50.239894 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:50.240323 (+ 429us) client_negotiation.cc:769] Sending connection context
0810 18:05:50.240555 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 18:05:50.240910 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":35,"spinlock_wait_cycles":4864,"thread_start_us":150,"threads_started":1}
I20260810 18:05:50.242188 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.228756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.28.83.125:42715)
0810 18:05:50.229139 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.229149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.229190 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:05:50.229316 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.229330 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.229584 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.229802 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.230306 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.231811 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.232866 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.233301 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.233513 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.235945 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.235955 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.235974 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.236010 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.238419 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.239183 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.239191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.239194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.239335 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.239736 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.239742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.239746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.240283 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.240442 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.241381 (+ 939us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.241580 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":280,"thread_start_us":146,"threads_started":1}
I20260810 18:05:50.244249 19732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:50.244433 19654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:50.245220 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.244985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55020)
0810 18:05:50.245090 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.245147 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 18:05:50.246071 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.245760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55032)
0810 18:05:50.245950 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.246008 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260810 18:05:50.246248 19654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:38729: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385150, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab.
Entry for principal kudu/127.28.83.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab.
I20260810 18:05:50.252539 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.124:38729
--webserver_interface=127.28.83.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab
--principal=kudu/127.28.83.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:33543,127.28.83.125:42715,127.28.83.124:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:05:50.253351 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.244877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42715 (local address 127.0.0.1:44112)
0810 18:05:50.245013 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.245028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:50.245111 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:50.245516 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:50.245519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:50.245716 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:50.245835 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.245842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.246797 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.246799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.247449 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.247459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.247940 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.247947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.248063 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.248515 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:50.248526 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:50.250117 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:50.251859 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.251862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.251863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.252059 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.252372 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.252375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.252376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.252451 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.252929 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:50.252931 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:50.253017 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 18:05:50.253104 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 18:05:50.253227 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 18:05:50.253621 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.244939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.28.83.125:42715)
0810 18:05:50.245219 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.245221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.245229 (+ 8us) server_negotiation.cc:407] Connection header received
0810 18:05:50.245250 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.245252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.245279 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.245359 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.245963 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.246690 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.247556 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.247833 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.247917 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.250264 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.250267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.250270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.250299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.251711 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.252161 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.252167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.252171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.252270 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.252558 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.252564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.252568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.252837 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.252943 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.253191 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.253340 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260810 18:05:50.254419 19732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:50.255271 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.254914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55036)
0810 18:05:50.255073 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.255183 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 18:05:50.255999 19683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
I20260810 18:05:50.257015 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.256798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55048)
0810 18:05:50.256918 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.256965 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 18:05:50.257202 19732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:38729: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
I20260810 18:05:50.277459 19654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } attempt: 1
I20260810 18:05:50.278378 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.278035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55050)
0810 18:05:50.278233 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.278303 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 18:05:50.279155 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.278962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55062)
0810 18:05:50.279060 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.279107 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 18:05:50.279363 19654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:38729: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
I20260810 18:05:50.306525 19732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } attempt: 1
I20260810 18:05:50.307433 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.307117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55076)
0810 18:05:50.307277 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.307349 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 18:05:50.308154 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.307936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55092)
0810 18:05:50.308043 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.308098 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 18:05:50.308302 19732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:38729: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
I20260810 18:05:50.354568 19654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } attempt: 2
I20260810 18:05:50.355502 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.355193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55094)
0810 18:05:50.355357 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.355425 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 18:05:50.356309 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.356080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55100)
0810 18:05:50.356197 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.356254 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:05:50.356438 19654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:38729: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
W20260810 18:05:50.373333 19738 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:05:50.373831 19738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:50.373898 19738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:50.373977 19738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 18:05:50.376479 19732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } attempt: 2
I20260810 18:05:50.377476 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.377065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55116)
0810 18:05:50.377256 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.377337 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 18:05:50.378793 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.378315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55118)
0810 18:05:50.378573 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.378660 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260810 18:05:50.379069 19732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:38729: Network error: Client connection negotiation failed: client connection to 127.28.83.124:38729: connect: Connection refused (error 111)
W20260810 18:05:50.379313 19738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:50.379382 19738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:50.379639 19738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:50.379688 19738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:50.379736 19738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:50.386440 19738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:33543,127.28.83.125:42715,127.28.83.124:38729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab
--principal=kudu/127.28.83.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.124:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.83.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:50.387687 19738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:50.388792 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:50.398094 19744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:50.398319 19738 server_base.cc:1034] running on GCE node
W20260810 18:05:50.398161 19747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:50.398101 19745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:50.399240 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:50.400460 19738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:50.401626 19738 hybrid_clock.cc:648] HybridClock initialized: now 1786385150401606 us; error 36 us; skew 500 ppm
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385150, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:50.404744 19738 init.cc:377] Logged in from keytab as kudu/127.28.83.124@KRBTEST.COM (short username kudu)
I20260810 18:05:50.405848 19738 webserver.cc:460] Webserver started at http://127.28.83.124:43325/ using document root <none> and password file <none>
I20260810 18:05:50.406448 19738 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:50.406517 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:50.406760 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:50.409037 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/data/instance:
uuid: "d8f8af5b644f4a95af75399d414dc0a6"
format_stamp: "Formatted at 2026-08-10 18:05:50 on dist-test-slave-srgv"
server_key: "ad451f3d7c630a739ebf7c5ed530e63a"
server_key_iv: "815dc49757c920be6b8d275e65dc0925"
server_key_version: "encryptionkey@0"
I20260810 18:05:50.409734 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/wal/instance:
uuid: "d8f8af5b644f4a95af75399d414dc0a6"
format_stamp: "Formatted at 2026-08-10 18:05:50 on dist-test-slave-srgv"
server_key: "ad451f3d7c630a739ebf7c5ed530e63a"
server_key_iv: "815dc49757c920be6b8d275e65dc0925"
server_key_version: "encryptionkey@0"
I20260810 18:05:50.413911 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:05:50.416602 19754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:50.418130 19738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:05:50.418329 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/wal
uuid: "d8f8af5b644f4a95af75399d414dc0a6"
format_stamp: "Formatted at 2026-08-10 18:05:50 on dist-test-slave-srgv"
server_key: "ad451f3d7c630a739ebf7c5ed530e63a"
server_key_iv: "815dc49757c920be6b8d275e65dc0925"
server_key_version: "encryptionkey@0"
I20260810 18:05:50.418475 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:50.432281 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:50.435922 19738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:50.436177 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:50.444286 19738 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.124:38729
I20260810 18:05:50.444583 19807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.124:38729 every 8 connection(s)
I20260810 18:05:50.445591 19738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
I20260810 18:05:50.448596 19808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:50.451435 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 19738
I20260810 18:05:50.451562 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786384863806381-29005-0/minicluster-data/master-2/wal/instance
I20260810 18:05:50.451882 29005 external_mini_cluster.cc:1428] Setting key 876f351756492059b4955674ff1acc10
I20260810 18:05:50.453078 19808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:50.455602 19654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } attempt: 3
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385149, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385150, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
I20260810 18:05:50.468508 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.455234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.28.83.126:33543)
0810 18:05:50.455372 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.455377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.455620 (+ 243us) server_negotiation.cc:407] Connection header received
0810 18:05:50.455949 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.455955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.456010 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.456093 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.457560 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.458073 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.459283 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.459517 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.459639 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.463854 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.463858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.463860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.463888 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.465923 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.466590 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.466593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.466596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.466694 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.467141 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.467147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.467148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.467538 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.467619 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.468178 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.468334 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:05:50.469100 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.456029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55140)
0810 18:05:50.456228 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.456243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:50.456311 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:50.456799 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:50.456802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:50.456977 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:50.457079 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.457088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.458854 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.458858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.459618 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.459630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.460474 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.460477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.460564 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.460997 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:50.461013 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:50.463015 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:50.466539 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.466544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.466547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.466853 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.467182 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.467185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.467187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.467290 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.468008 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:50.468011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:50.468114 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 18:05:50.468232 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 18:05:50.468941 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":557,"spinlock_wait_cycles":1920}
I20260810 18:05:50.469746 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.456290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.28.83.124:38729)
0810 18:05:50.456549 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.456557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.456570 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:05:50.456603 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.456605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.456638 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.456767 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.457259 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.458726 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.459774 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.460329 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.460467 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.463179 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.463188 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.463206 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.463244 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.466403 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.466974 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.466980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.466982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.467080 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.467383 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.467386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.467388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.467898 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.468044 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.468412 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.468733 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":200,"spinlock_wait_cycles":13568,"thread_start_us":108,"threads_started":1}
I20260810 18:05:50.469745 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.453476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.28.83.124:38729)
0810 18:05:50.454013 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.454031 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.454070 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:05:50.454780 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.454798 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.455230 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.456542 (+ 1312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.457129 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.458726 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.460435 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.460773 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.460888 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.464771 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.464776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.464780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.464813 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.467208 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.467732 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.467735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.467737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.467811 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.468141 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.468144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.468146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.468332 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.469108 (+ 776us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.469190 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.469300 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":339,"server-negotiator.queue_time_us":375,"spinlock_wait_cycles":1536,"thread_start_us":179,"threads_started":1}
I20260810 18:05:50.469745 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.455079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:33543 (local address 127.0.0.1:55730)
0810 18:05:50.455495 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.455550 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:50.455713 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:50.456204 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:50.456217 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:50.456717 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:50.457390 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.457448 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.458195 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.458198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.459155 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.459177 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.459700 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.459706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.459944 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.460890 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:50.460915 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:50.463713 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:50.466069 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.466079 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.466093 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.466464 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.466798 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.466801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.466803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.467014 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.467714 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:50.467723 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:50.467919 (+ 196us) client_negotiation.cc:769] Sending connection context
0810 18:05:50.468125 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 18:05:50.468525 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":53,"spinlock_wait_cycles":6656,"thread_start_us":103,"threads_started":1}
I20260810 18:05:50.470248 19732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } attempt: 3
I20260810 18:05:50.472744 19808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:50.474870 19654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9: Bootstrap starting.
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385150, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
I20260810 18:05:50.477861 19654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:50.478847 19654 log.cc:824] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:50.481066 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.470787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55142)
0810 18:05:50.470935 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.470952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:50.471040 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:50.471614 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:50.471617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:50.471802 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:50.471891 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.471896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.472871 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.472875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.473704 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.473720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.474216 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.474220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.474304 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.474859 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:50.474881 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:50.476460 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:50.479443 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.479446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.479447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.479665 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.479991 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.479994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.479996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.480084 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.480533 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:50.480536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:50.480628 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 18:05:50.480720 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:05:50.480869 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 18:05:50.481748 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.471010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.28.83.124:38729)
0810 18:05:50.471396 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.471400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.471429 (+ 29us) server_negotiation.cc:407] Connection header received
0810 18:05:50.471462 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.471465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.471513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.471590 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.472010 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.472755 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.473829 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.474095 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.474209 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.476615 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.476618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.476620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.476646 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.479329 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.479787 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.479790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.479792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.479885 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.480188 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.480193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.480195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.480410 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.481273 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.481362 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.481531 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260810 18:05:50.483424 19654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9: No bootstrap required, opened a new log
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385150, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:50.485045 19732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa: Bootstrap starting.
I20260810 18:05:50.488054 19732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:50.487883 19654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } }
I20260810 18:05:50.488139 19654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88166728902a4ef6a672ca8074722ee9, State: Initialized, Role: FOLLOWER
I20260810 18:05:50.488306 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.473305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:42715 (local address 127.0.0.1:44114)
0810 18:05:50.473489 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.473503 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:50.473577 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:50.477852 (+ 4275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:50.477856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:50.478039 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:50.478195 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.478204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.479471 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.479476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.480331 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.480342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.481023 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.481026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.481141 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.482149 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:50.482199 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:50.483920 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:50.486173 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.486182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.486187 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.486483 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.486850 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.486856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.486860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.486997 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.487687 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:50.487690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:50.487807 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 18:05:50.487920 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 18:05:50.488095 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 18:05:50.488505 19654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } }
I20260810 18:05:50.489025 19732 log.cc:824] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:50.489401 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.473373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.28.83.125:42715)
0810 18:05:50.477611 (+ 4238us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.477614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.477627 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:05:50.477660 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.477662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.477729 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.477814 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.478331 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.479073 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.480453 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.480803 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.480897 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.484115 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.484119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.484122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.484148 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.486025 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.486619 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.486625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.486628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.486722 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.487207 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.487210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.487212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.487525 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.487632 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.489028 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.489247 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4138}
I20260810 18:05:50.490003 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.478612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.28.83.126:33543)
0810 18:05:50.478741 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.478746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.478760 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:05:50.478885 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.478888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.478937 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.479013 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.479938 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.480636 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.481787 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.482076 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.482198 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.485775 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.485780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.485784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.485815 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.487574 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.488112 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.488115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.488117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.488204 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.488531 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.488537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.488541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.488802 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.488981 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.489726 (+ 745us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.489853 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:05:50.490724 19808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:50.490650 19654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:50.491644 19732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa: No bootstrap required, opened a new log
W20260810 18:05:50.496698 19827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:50.496788 19827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385150, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
Aug 10 18:05:50 dist-test-slave-srgv krb5kdc[19570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
I20260810 18:05:50.495946 19585 master_runner.cc:186] Error getting master registration for 127.28.83.125:42715: OK, instance_id { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" instance_seqno: 1786385150213972 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:05:50.500119 19585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:05:50.500515 19732 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } }
I20260810 18:05:50.500725 19732 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c0b3b2e1ed44f4acbb247b7cfafcaa, State: Initialized, Role: FOLLOWER
I20260810 18:05:50.501010 19732 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } }
I20260810 18:05:50.501528 19829 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.501667 19829 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:50.502553 19732 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:50.503451 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.491320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:38729 (local address 127.0.0.1:55156)
0810 18:05:50.491991 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:50.492004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:50.492094 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:50.492819 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:50.492823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:50.493005 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:50.493095 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.493102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.494210 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.494217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.495060 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:50.495069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.495672 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.495685 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:50.495762 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.496437 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:50.496454 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:50.499319 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:50.501548 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.501552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.501554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.501817 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.502149 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:50.502152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:50.502154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:50.502276 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:50.502835 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:50.502838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:50.502933 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 18:05:50.503055 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 18:05:50.503232 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602}
I20260810 18:05:50.503582 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.491874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.28.83.124:38729)
0810 18:05:50.492580 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.492584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.492597 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:05:50.492632 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.492636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.492695 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.493198 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.493272 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.494103 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.495205 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.495547 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.496023 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.499463 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.499467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.499472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.499502 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.501380 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.501941 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.501944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.501946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.502025 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.502404 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.502408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.502409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.502737 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.502848 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.503130 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.503315 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":627}
I20260810 18:05:50.507048 19808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6: Bootstrap starting.
W20260810 18:05:50.508212 19841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:50.508298 19841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:05:50.504933 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.491712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.28.83.125:42715)
0810 18:05:50.492254 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.492257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.492268 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:05:50.492298 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.492300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.492332 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.492422 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.493243 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.494085 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.495234 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.495561 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.498089 (+ 2528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.499028 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.499035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.499038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.499064 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.502120 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.502674 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.502677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.502679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.502772 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.503125 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.503128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.503130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.503367 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.504575 (+ 1208us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.504663 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.504792 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463}
I20260810 18:05:50.509917 19808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:50.510903 19808 log.cc:824] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:50.513324 19808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6: No bootstrap required, opened a new log
I20260810 18:05:50.510818 19661 master_runner.cc:186] Error getting master registration for 127.28.83.124:38729: OK, instance_id { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" instance_seqno: 1786385150441915 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:05:50.514422 19661 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:05:50.517710 19815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.517925 19815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:50.518405 19808 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } }
I20260810 18:05:50.518842 19808 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f8af5b644f4a95af75399d414dc0a6, State: Initialized, Role: FOLLOWER
I20260810 18:05:50.519241 19808 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } }
I20260810 18:05:50.520823 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:50.520952 19808 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:50.522372 19628 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "88166728902a4ef6a672ca8074722ee9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55738
I20260810 18:05:50.522751 19628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:05:50.522891 19628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:50.524490 19628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } }
I20260810 18:05:50.526588 19781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88166728902a4ef6a672ca8074722ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d8f8af5b644f4a95af75399d414dc0a6"
I20260810 18:05:50.526885 19781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [term 0 FOLLOWER]: Advancing to term 1
W20260810 18:05:50.528227 19856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:50.528333 19856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:05:50.528482 19781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88166728902a4ef6a672ca8074722ee9 in term 1.
I20260810 18:05:50.528679 19844 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.528823 19844 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:50.529021 19706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88166728902a4ef6a672ca8074722ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa"
I20260810 18:05:50.529314 19706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:50.529747 19605 leader_election.cc:302] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88166728902a4ef6a672ca8074722ee9, d8f8af5b644f4a95af75399d414dc0a6; no voters:
I20260810 18:05:50.530113 19815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:50.530344 19815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [term 1 LEADER]: Becoming Leader. State: Replica: 88166728902a4ef6a672ca8074722ee9, State: Running, Role: LEADER
I20260810 18:05:50.530710 19815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } }
I20260810 18:05:50.532527 19706 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88166728902a4ef6a672ca8074722ee9 in term 1.
I20260810 18:05:50.532649 19815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88166728902a4ef6a672ca8074722ee9. Latest consensus state: current_term: 1 leader_uuid: "88166728902a4ef6a672ca8074722ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.532778 19815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:50.533486 19860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:50.534508 19860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:50.538125 19738 master_runner.cc:263] No leader master found from master d8f8af5b644f4a95af75399d414dc0a6
I20260810 18:05:50.538218 19738 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:05:50.538291 19628 leader_election.cc:288] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [CANDIDATE]: Term 1 election: Requested vote from peers c0c0b3b2e1ed44f4acbb247b7cfafcaa (127.28.83.125:42715), d8f8af5b644f4a95af75399d414dc0a6 (127.28.83.124:38729)
I20260810 18:05:50.542312 19781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [term 1 FOLLOWER]: Refusing update from remote peer 88166728902a4ef6a672ca8074722ee9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:50.543139 19858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:50.547324 19844 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88166728902a4ef6a672ca8074722ee9. Latest consensus state: current_term: 1 leader_uuid: "88166728902a4ef6a672ca8074722ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.547468 19844 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:50.548621 19706 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [term 1 FOLLOWER]: Refusing update from remote peer 88166728902a4ef6a672ca8074722ee9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:50.549253 19858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:50.553222 19829 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88166728902a4ef6a672ca8074722ee9. Latest consensus state: current_term: 1 leader_uuid: "88166728902a4ef6a672ca8074722ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.553387 19829 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:50.554546 19858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88166728902a4ef6a672ca8074722ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.554679 19858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:50.557847 19860 catalog_manager.cc:1347] Generated new cluster ID: 943d955fbb4546b1b3e696f6a8c124e8
I20260810 18:05:50.557919 19860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:50.564934 19868 mvcc.cc:205] Tried to move back new op lower bound from 7317033576613101568 to 7317033576577478656. Current Snapshot: MvccSnapshot[applied={T|T < 7317033576613101568}]
I20260810 18:05:50.565740 19815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88166728902a4ef6a672ca8074722ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.565907 19815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:50.567970 19829 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88166728902a4ef6a672ca8074722ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.568114 19829 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:50.568674 19870 mvcc.cc:205] Tried to move back new op lower bound from 7317033576613101568 to 7317033576577478656. Current Snapshot: MvccSnapshot[applied={T|T < 7317033576613101568}]
I20260810 18:05:50.569736 19844 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88166728902a4ef6a672ca8074722ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88166728902a4ef6a672ca8074722ee9" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 33543 } } peers { permanent_uuid: "c0c0b3b2e1ed44f4acbb247b7cfafcaa" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 42715 } } peers { permanent_uuid: "d8f8af5b644f4a95af75399d414dc0a6" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 38729 } } }
I20260810 18:05:50.569883 19844 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:50.580462 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.555883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.28.83.125:42715)
0810 18:05:50.555993 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.555997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.561679 (+ 5682us) server_negotiation.cc:407] Connection header received
0810 18:05:50.561870 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.561873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.561923 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.561973 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.562897 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.563683 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.565711 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.566040 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.566120 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.574036 (+ 7916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.574042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.574048 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.574081 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.576318 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.578932 (+ 2614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.578936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.578939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.579081 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.579400 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.579403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.579405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.579643 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.579779 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.580146 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.580291 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:05:50.580885 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.541605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.28.83.124:38729)
0810 18:05:50.541735 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.541739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.557654 (+ 15915us) server_negotiation.cc:407] Connection header received
0810 18:05:50.558939 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.558942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.558991 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.559048 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.562891 (+ 3843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.563681 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.566033 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.566361 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.566463 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.573026 (+ 6563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.573030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.573034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.573072 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.578181 (+ 5109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.579322 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.579325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.579327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.579414 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.579678 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.579681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.579683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.579883 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.579985 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.580592 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.580728 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:05:50.582301 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:50.540857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.28.83.126:33543)
0810 18:05:50.540972 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.540977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.556868 (+ 15891us) server_negotiation.cc:407] Connection header received
0810 18:05:50.557136 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.557139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.557189 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.557242 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.560836 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.561624 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.563909 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:50.564263 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:50.564346 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:50.573288 (+ 8942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:50.573293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:50.573296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:50.573331 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:50.577157 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.578947 (+ 1790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.578951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.578953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.579048 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:50.581287 (+ 2239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:50.581290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:50.581292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:50.581530 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:50.581668 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:50.582024 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 18:05:50.582152 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:05:50.596357 19860 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:50.600680 19860 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:50.628612 19860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88166728902a4ef6a672ca8074722ee9: Generated new TSK 0
I20260810 18:05:50.629770 19860 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:05:50.668846 19658 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.665037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.28.83.126:33543)
0810 18:05:50.665195 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.665200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.667394 (+ 2194us) server_negotiation.cc:407] Connection header received
0810 18:05:50.667841 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.667845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.667904 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.667960 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.668692 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55756: BlockingRecv error: recv got EOF from 127.0.0.1:55756 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 18:05:50.670352 19813 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.665935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.28.83.124:38729)
0810 18:05:50.666073 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.666079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.667748 (+ 1669us) server_negotiation.cc:407] Connection header received
0810 18:05:50.669492 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.669495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.669555 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.669610 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.670217 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55170: BlockingRecv error: recv got EOF from 127.0.0.1:55170 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 18:05:50.672294 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.666731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.28.83.125:42715)
0810 18:05:50.666856 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.666860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.671052 (+ 4192us) server_negotiation.cc:407] Connection header received
0810 18:05:50.671458 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.671461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.671519 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.671575 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.672154 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44142: BlockingRecv error: recv got EOF from 127.0.0.1:44142 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 18:05:50.677920 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.675971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.28.83.125:42715)
0810 18:05:50.676103 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.676108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.676683 (+ 575us) server_negotiation.cc:407] Connection header received
0810 18:05:50.676929 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.676932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.677015 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.677068 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.677800 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44146: BlockingRecv error: recv got EOF from 127.0.0.1:44146 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 18:05:50.684015 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.682248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.28.83.125:42715)
0810 18:05:50.682383 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.682387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.682949 (+ 562us) server_negotiation.cc:407] Connection header received
0810 18:05:50.683174 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.683181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.683236 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.683291 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.683889 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44156: BlockingRecv error: recv got EOF from 127.0.0.1:44156 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 18:05:50.690680 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.688782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.28.83.125:42715)
0810 18:05:50.688901 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.688905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.689575 (+ 670us) server_negotiation.cc:407] Connection header received
0810 18:05:50.689810 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.689813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.689873 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.689932 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.690553 (+ 621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44172: BlockingRecv error: recv got EOF from 127.0.0.1:44172 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 18:05:50.699497 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.697569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.28.83.125:42715)
0810 18:05:50.697746 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.697751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.698380 (+ 629us) server_negotiation.cc:407] Connection header received
0810 18:05:50.698613 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.698617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.698675 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.698735 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.699374 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44182: BlockingRecv error: recv got EOF from 127.0.0.1:44182 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260810 18:05:50.713290 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.711456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.28.83.125:42715)
0810 18:05:50.711601 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.711606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.712247 (+ 641us) server_negotiation.cc:407] Connection header received
0810 18:05:50.712462 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.712465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.712521 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.712575 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.713168 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44192: BlockingRecv error: recv got EOF from 127.0.0.1:44192 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 18:05:50.733305 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.732105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.28.83.125:42715)
0810 18:05:50.732252 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.732257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.732412 (+ 155us) server_negotiation.cc:407] Connection header received
0810 18:05:50.732523 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.732527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.732597 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.732711 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.733170 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44208: BlockingRecv error: recv got EOF from 127.0.0.1:44208 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 18:05:50.769045 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.767805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.28.83.125:42715)
0810 18:05:50.767971 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.767976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.768109 (+ 133us) server_negotiation.cc:407] Connection header received
0810 18:05:50.768218 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.768221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.768280 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.768391 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.768922 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44218: BlockingRecv error: recv got EOF from 127.0.0.1:44218 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 18:05:50.836421 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.835315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.28.83.125:42715)
0810 18:05:50.835475 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.835479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.835609 (+ 130us) server_negotiation.cc:407] Connection header received
0810 18:05:50.835744 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.835746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.835795 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.835882 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.836320 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44232: BlockingRecv error: recv got EOF from 127.0.0.1:44232 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 18:05:50.967962 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:50.966691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.28.83.125:42715)
0810 18:05:50.966857 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:50.966863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:50.967016 (+ 153us) server_negotiation.cc:407] Connection header received
0810 18:05:50.967215 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:50.967218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:50.967270 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:50.967353 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:50.967823 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44248: BlockingRecv error: recv got EOF from 127.0.0.1:44248 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 18:05:51.227583 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:51.226284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.28.83.125:42715)
0810 18:05:51.226431 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:51.226439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:51.226649 (+ 210us) server_negotiation.cc:407] Connection header received
0810 18:05:51.226847 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:51.226850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:51.226898 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:51.226989 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:51.227464 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44256: BlockingRecv error: recv got EOF from 127.0.0.1:44256 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:05:51.509766 19841 catalog_manager.cc:1259] Loaded cluster ID: 943d955fbb4546b1b3e696f6a8c124e8
I20260810 18:05:51.509866 19841 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa: loading cluster ID for follower catalog manager: success
I20260810 18:05:51.511644 19841 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa: acquiring CA information for follower catalog manager: success
I20260810 18:05:51.512975 19841 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0c0b3b2e1ed44f4acbb247b7cfafcaa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:05:51.529754 19856 catalog_manager.cc:1259] Loaded cluster ID: 943d955fbb4546b1b3e696f6a8c124e8
I20260810 18:05:51.529884 19856 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6: loading cluster ID for follower catalog manager: success
I20260810 18:05:51.531555 19856 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6: acquiring CA information for follower catalog manager: success
I20260810 18:05:51.532778 19856 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8f8af5b644f4a95af75399d414dc0a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:05:51.745756 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:51.741844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.28.83.125:42715)
0810 18:05:51.742121 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:51.742125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:51.742142 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:05:51.742270 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:51.742272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:51.742402 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:51.742505 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:51.742784 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.743309 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.744196 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.744447 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.744534 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:51.744820 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:51.745262 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:51.745359 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:51.745489 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 18:05:51.745547 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260810 18:05:51.752668 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:51.749109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.28.83.124:38729)
0810 18:05:51.749367 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:51.749371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:51.749385 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:05:51.749527 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:51.749529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:51.749650 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:51.749757 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:51.749960 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.750523 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.751328 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.751546 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.751623 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:51.751887 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:51.752325 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:51.752415 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:51.752493 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 18:05:51.752544 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":118,"threads_started":1}
I20260810 18:05:51.760318 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:51.756195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.28.83.124:38729)
0810 18:05:51.756316 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:51.756320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:51.756741 (+ 421us) server_negotiation.cc:407] Connection header received
0810 18:05:51.756935 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:51.756938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:51.757034 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:51.757183 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:51.757450 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.758150 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.759081 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.759348 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.759449 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:51.759730 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:51.759804 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:51.759876 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:51.759975 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 18:05:51.760029 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:05:51.760318 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:51.755837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.28.83.125:42715)
0810 18:05:51.755981 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:51.755986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:51.756309 (+ 323us) server_negotiation.cc:407] Connection header received
0810 18:05:51.756493 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:51.756497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:51.756557 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:51.756661 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:51.756917 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.757675 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.758901 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.759231 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.759341 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:51.759666 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:51.759759 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:51.759830 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:51.760125 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 18:05:51.760196 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:05:51.761586 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:51.755763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.28.83.126:33543)
0810 18:05:51.756069 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:51.756076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:51.756117 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:05:51.756410 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:51.756415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:51.756631 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:51.756754 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:51.757023 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.757928 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.759068 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:51.759416 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:51.759511 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:51.760146 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:51.761129 (+ 983us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:51.761224 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:51.761350 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 18:05:51.761404 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
I20260810 18:05:51.764549 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 19585
I20260810 18:05:51.772599 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 19661
I20260810 18:05:51.779881 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 19738
2026-08-10T18:05:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3175 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:05:51 dist-test-slave-srgv krb5kdc[19961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:05:51 dist-test-slave-srgv krb5kdc[19961](info): set up 2 sockets
Aug 10 18:05:51 dist-test-slave-srgv krb5kdc[19961](info): commencing operation
krb5kdc: starting...
W20260810 18:05:52.848817 29005 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/test-user.keytab.
Aug 10 18:05:52 dist-test-slave-srgv krb5kdc[19961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:05:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
Entry for principal kudu/127.28.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab.
I20260810 18:05:52.979955 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43309
--webserver_interface=127.28.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:43309,127.28.83.125:43307,127.28.83.124:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:05:53.094620 19975 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:05:53.094949 19975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:53.094973 19975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:53.095014 19975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:05:53.099128 19975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:53.099198 19975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:53.099218 19975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:53.099236 19975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:53.099251 19975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:53.104843 19975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:43309,127.28.83.125:43307,127.28.83.124:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.126.keytab
--principal=kudu/127.28.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.126:43309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:53.106011 19975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:53.107007 19975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:53.115926 19984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:53.115902 19981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:53.115902 19986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:53.116238 19975 server_base.cc:1034] running on GCE node
I20260810 18:05:53.116933 19975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:53.118180 19975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:53.119383 19975 hybrid_clock.cc:648] HybridClock initialized: now 1786385153119364 us; error 36 us; skew 500 ppm
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:53.122262 19975 init.cc:377] Logged in from keytab as kudu/127.28.83.126@KRBTEST.COM (short username kudu)
I20260810 18:05:53.123206 19975 webserver.cc:460] Webserver started at http://127.28.83.126:42907/ using document root <none> and password file <none>
I20260810 18:05:53.123791 19975 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:53.123888 19975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:53.124151 19975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:53.126332 19975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/data/instance:
uuid: "8263608987fd4c3080904413ceb76bca"
format_stamp: "Formatted at 2026-08-10 18:05:53 on dist-test-slave-srgv"
server_key: "ea75b33df7c8b261193ae3aea7392458"
server_key_iv: "7baabc89da1aa4b2341567057043bc19"
server_key_version: "encryptionkey@0"
I20260810 18:05:53.126947 19975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/wal/instance:
uuid: "8263608987fd4c3080904413ceb76bca"
format_stamp: "Formatted at 2026-08-10 18:05:53 on dist-test-slave-srgv"
server_key: "ea75b33df7c8b261193ae3aea7392458"
server_key_iv: "7baabc89da1aa4b2341567057043bc19"
server_key_version: "encryptionkey@0"
I20260810 18:05:53.130904 19975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:05:53.133365 19994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:53.134653 19975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:05:53.134824 19975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/wal
uuid: "8263608987fd4c3080904413ceb76bca"
format_stamp: "Formatted at 2026-08-10 18:05:53 on dist-test-slave-srgv"
server_key: "ea75b33df7c8b261193ae3aea7392458"
server_key_iv: "7baabc89da1aa4b2341567057043bc19"
server_key_version: "encryptionkey@0"
I20260810 18:05:53.134951 19975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:53.145150 19975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:53.148138 19975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:53.148365 19975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:53.155831 19975 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.126:43309
I20260810 18:05:53.156139 20050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.126:43309 every 8 connection(s)
I20260810 18:05:53.156854 19975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/data/info.pb
I20260810 18:05:53.157590 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 19975
I20260810 18:05:53.157728 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-0/wal/instance
I20260810 18:05:53.158037 29005 external_mini_cluster.cc:1428] Setting key c05f9917dde2984b3310c9848d130e72
I20260810 18:05:53.160372 20051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:53.164350 20051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.83.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.125@KRBTEST.COM" created.
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:53.181691 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.166839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43309 (local address 127.0.0.1:33952)
0810 18:05:53.167552 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.167597 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:53.168493 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:53.169310 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:53.169319 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:53.169746 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:53.169995 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.170009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.171579 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.171585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.172870 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.172883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.173605 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.173613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.173793 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.174458 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:53.174479 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:53.176462 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:53.178617 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.178630 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.178657 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.178996 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.179295 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.179298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.179299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.179412 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.180114 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:53.180120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:53.180255 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 18:05:53.180407 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 18:05:53.180811 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":122,"thread_start_us":169,"threads_started":1}
I20260810 18:05:53.181696 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.166839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.28.83.126:43309)
0810 18:05:53.167324 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.167343 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.167733 (+ 390us) server_negotiation.cc:407] Connection header received
0810 18:05:53.168841 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.168847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.169098 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.169266 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.170156 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.171443 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.173004 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.173379 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.173571 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.176636 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.176642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.176653 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.176675 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.178488 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.179099 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.179104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.179105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.179208 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.179522 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.179525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.179526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.180004 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.180114 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.180518 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.180738 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":344,"spinlock_wait_cycles":18688,"thread_start_us":154,"threads_started":1}
I20260810 18:05:53.184229 20051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:53.185057 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.184786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:43307 (local address 127.0.0.1:37698)
0810 18:05:53.184906 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.184990 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 18:05:53.185336 19996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
I20260810 18:05:53.186395 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.186191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:43307 (local address 127.0.0.1:37712)
0810 18:05:53.186295 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.186342 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 18:05:53.186573 20051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.125:43307: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab.
Entry for principal kudu/127.28.83.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab.
I20260810 18:05:53.189381 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.125:43307
--webserver_interface=127.28.83.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab
--principal=kudu/127.28.83.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:43309,127.28.83.125:43307,127.28.83.124:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:05:53.235868 20051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } attempt: 1
I20260810 18:05:53.236910 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.236594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:43307 (local address 127.0.0.1:37722)
0810 18:05:53.236763 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.236830 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 18:05:53.237766 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.237535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:43307 (local address 127.0.0.1:37736)
0810 18:05:53.237654 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.237709 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 18:05:53.237900 20051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.125:43307: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
I20260810 18:05:53.306087 20051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } attempt: 2
W20260810 18:05:53.306381 20058 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:05:53.306731 20058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:53.306809 20058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:53.306895 20058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 18:05:53.307314 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.306991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:43307 (local address 127.0.0.1:37742)
0810 18:05:53.307168 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.307236 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 18:05:53.308051 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.307824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:43307 (local address 127.0.0.1:37752)
0810 18:05:53.307936 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.307988 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:05:53.308177 20051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.125:43307: Network error: Client connection negotiation failed: client connection to 127.28.83.125:43307: connect: Connection refused (error 111)
W20260810 18:05:53.311383 20058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:53.311473 20058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:53.311513 20058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:53.311573 20058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:53.311612 20058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:53.318434 20058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:43309,127.28.83.125:43307,127.28.83.124:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.125.keytab
--principal=kudu/127.28.83.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.125:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.83.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:53.319634 20058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:53.320714 20058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:53.329521 20063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:53.329622 20058 server_base.cc:1034] running on GCE node
W20260810 18:05:53.329533 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:53.329829 20064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:53.330432 20058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:53.331568 20058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:53.332736 20058 hybrid_clock.cc:648] HybridClock initialized: now 1786385153332725 us; error 31 us; skew 500 ppm
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:53.335667 20058 init.cc:377] Logged in from keytab as kudu/127.28.83.125@KRBTEST.COM (short username kudu)
I20260810 18:05:53.336603 20058 webserver.cc:460] Webserver started at http://127.28.83.125:46385/ using document root <none> and password file <none>
I20260810 18:05:53.337126 20058 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:53.337186 20058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:53.337370 20058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:53.339679 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/data/instance:
uuid: "c1b42a1c198544c9949527df17a20039"
format_stamp: "Formatted at 2026-08-10 18:05:53 on dist-test-slave-srgv"
server_key: "9cdc3a720638b30692251c0398d75af1"
server_key_iv: "453b031fefaf11c7f18529e6d8cc5788"
server_key_version: "encryptionkey@0"
I20260810 18:05:53.340265 20058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/wal/instance:
uuid: "c1b42a1c198544c9949527df17a20039"
format_stamp: "Formatted at 2026-08-10 18:05:53 on dist-test-slave-srgv"
server_key: "9cdc3a720638b30692251c0398d75af1"
server_key_iv: "453b031fefaf11c7f18529e6d8cc5788"
server_key_version: "encryptionkey@0"
I20260810 18:05:53.344118 20058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:05:53.346591 20073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:53.347870 20058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:05:53.348022 20058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/wal
uuid: "c1b42a1c198544c9949527df17a20039"
format_stamp: "Formatted at 2026-08-10 18:05:53 on dist-test-slave-srgv"
server_key: "9cdc3a720638b30692251c0398d75af1"
server_key_iv: "453b031fefaf11c7f18529e6d8cc5788"
server_key_version: "encryptionkey@0"
I20260810 18:05:53.348143 20058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:53.376770 20058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:53.380425 20058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:53.380697 20058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:53.388473 20058 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.125:43307
I20260810 18:05:53.388769 20127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.125:43307 every 8 connection(s)
I20260810 18:05:53.389631 20058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/data/info.pb
I20260810 18:05:53.392500 20128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:53.396327 20128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:53.397223 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 20058
I20260810 18:05:53.397380 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-1/wal/instance
I20260810 18:05:53.397715 29005 external_mini_cluster.cc:1428] Setting key b6f610582c12992cb80f3629b2fd70db
I20260810 18:05:53.399348 20051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } attempt: 3
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.83.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.83.124@KRBTEST.COM" created.
I20260810 18:05:53.411954 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.399775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:43307 (local address 127.0.0.1:37760)
0810 18:05:53.399953 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.399970 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:53.400052 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:53.401036 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:53.401040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:53.401243 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:53.401375 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.401384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.403058 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.403063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.403741 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.403749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.404532 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.404537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.404821 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.405481 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:53.405497 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:53.407188 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:53.409570 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.409575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.409578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.409936 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.410368 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.410372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.410374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.410472 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.411258 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:53.411264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:53.411377 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 18:05:53.411484 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 18:05:53.411673 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":58}
I20260810 18:05:53.412405 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.398380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.28.83.126:43309)
0810 18:05:53.398572 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.398580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.399449 (+ 869us) server_negotiation.cc:407] Connection header received
0810 18:05:53.400676 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.400679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.400738 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.400844 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.401936 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.402711 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.404398 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.404876 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.404992 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.407772 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.407777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.407782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.407809 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.410368 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.411004 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.411010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.411015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.411106 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.411429 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.411433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.411435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.411616 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.411729 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.412144 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.412265 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":59}
I20260810 18:05:53.413311 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.398731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43309 (local address 127.0.0.1:33954)
0810 18:05:53.399274 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.399338 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:53.400460 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:53.400900 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:53.400909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:53.401389 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:53.401779 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.401798 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.402844 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.402851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.404259 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.404273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.405040 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.405045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.405177 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.405767 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:53.405783 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:53.407615 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:53.410467 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.410474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.410479 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.410866 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.411216 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.411220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.411224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.411315 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.411727 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:53.411735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:53.411903 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 18:05:53.412777 (+ 874us) client_negotiation.cc:240] Negotiation successful
0810 18:05:53.412965 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":45,"spinlock_wait_cycles":10624,"thread_start_us":123,"threads_started":1}
I20260810 18:05:53.413311 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.399920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.28.83.125:43307)
0810 18:05:53.400346 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.400358 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.400404 (+ 46us) server_negotiation.cc:407] Connection header received
0810 18:05:53.400564 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.400585 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.400852 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.401052 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.401540 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.402949 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.403858 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.404388 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.404555 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.407387 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.407395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.407412 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.407446 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.409399 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.410109 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.410114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.410120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.410266 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.410580 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.410585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.410590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.411144 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.411316 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.411910 (+ 594us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.412231 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":303,"thread_start_us":132,"threads_started":1}
I20260810 18:05:53.414863 20128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:53.415696 20051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:53.416404 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.416117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38236)
0810 18:05:53.416250 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.416328 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 18:05:53.417117 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.416863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38250)
0810 18:05:53.417015 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.417068 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 18:05:53.417294 20051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43985: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
I20260810 18:05:53.423748 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.415390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:43307 (local address 127.0.0.1:37776)
0810 18:05:53.415481 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.415498 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:53.415570 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:53.415943 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:53.415946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:53.416112 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:53.416219 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.416225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.417369 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.417372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.418020 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.418029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.418513 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.418521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.418687 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.419220 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:53.419231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:53.420446 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:53.422275 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.422281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.422285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.422473 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.422743 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.422746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.422748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.422828 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.423200 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:53.423203 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:53.423290 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 18:05:53.423390 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 18:05:53.423528 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":35}
I20260810 18:05:53.423748 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.415428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.28.83.125:43307)
0810 18:05:53.415746 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.415749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.415759 (+ 10us) server_negotiation.cc:407] Connection header received
0810 18:05:53.415809 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.415814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.415859 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.415933 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.416350 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.417241 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.418156 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.418405 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.418520 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.420570 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.420574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.420576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.420598 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.422156 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.422566 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.422572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.422574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.422647 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.422912 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.422917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.422919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.423091 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.423201 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.423447 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.423580 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":254}
I20260810 18:05:53.424870 20128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:53.425647 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.425315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38262)
0810 18:05:53.425472 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.425569 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 18:05:53.426237 20077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
I20260810 18:05:53.427301 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.427051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38268)
0810 18:05:53.427153 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.427203 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 18:05:53.427451 20128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43985: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.83.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab.
Entry for principal kudu/127.28.83.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab.
I20260810 18:05:53.428714 29005 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EWyHF/build/debug/bin/kudu
/tmp/dist-test-task5EWyHF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.124:43985
--webserver_interface=127.28.83.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab
--principal=kudu/127.28.83.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.83.84:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.83.126:43309,127.28.83.125:43307,127.28.83.124:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:05:53.448480 20051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } attempt: 1
I20260810 18:05:53.449470 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.449089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38282)
0810 18:05:53.449271 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.449345 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 18:05:53.450268 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.450013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38294)
0810 18:05:53.450150 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.450204 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 18:05:53.450420 20051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43985: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
I20260810 18:05:53.476751 20128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } attempt: 1
I20260810 18:05:53.477746 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.477350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38310)
0810 18:05:53.477572 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.477656 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 18:05:53.478647 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.478418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38312)
0810 18:05:53.478534 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.478588 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:05:53.478817 20128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43985: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
I20260810 18:05:53.525651 20051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } attempt: 2
I20260810 18:05:53.526585 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.526227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38322)
0810 18:05:53.526411 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.526485 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 18:05:53.527549 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.527301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38334)
0810 18:05:53.527422 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.527475 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 18:05:53.527699 20051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43985: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
W20260810 18:05:53.546214 20137 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:05:53.546569 20137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:05:53.546627 20137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:05:53.546710 20137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 18:05:53.547008 20128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } attempt: 2
I20260810 18:05:53.547927 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.547605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38344)
0810 18:05:53.547774 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.547842 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 18:05:53.548723 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.548512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38354)
0810 18:05:53.548613 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.548659 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 18:05:53.548866 20128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.83.124:43985: Network error: Client connection negotiation failed: client connection to 127.28.83.124:43985: connect: Connection refused (error 111)
W20260810 18:05:53.551689 20137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:05:53.551750 20137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:05:53.551797 20137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:05:53.551841 20137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:05:53.551891 20137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:05:53.557811 20137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.83.84:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.83.126:43309,127.28.83.125:43307,127.28.83.124:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/krb5kdc/kudu_127.28.83.124.keytab
--principal=kudu/127.28.83.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.83.124:43985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.83.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:05:53.559002 20137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:05:53.560082 20137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:05:53.569475 20145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:53.569393 20143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:05:53.569487 20142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:05:53.569869 20137 server_base.cc:1034] running on GCE node
I20260810 18:05:53.570456 20137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:05:53.571586 20137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:05:53.572762 20137 hybrid_clock.cc:648] HybridClock initialized: now 1786385153572744 us; error 29 us; skew 500 ppm
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:05:53.575819 20137 init.cc:377] Logged in from keytab as kudu/127.28.83.124@KRBTEST.COM (short username kudu)
I20260810 18:05:53.576776 20137 webserver.cc:460] Webserver started at http://127.28.83.124:39253/ using document root <none> and password file <none>
I20260810 18:05:53.577369 20137 fs_manager.cc:359] Metadata directory not provided
I20260810 18:05:53.577517 20137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:05:53.577750 20137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:05:53.579937 20137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/data/instance:
uuid: "5161b8ecb2bf4557a3e0b2db6683ce47"
format_stamp: "Formatted at 2026-08-10 18:05:53 on dist-test-slave-srgv"
server_key: "030a7b59556f5cf6cf7507ac04532196"
server_key_iv: "bc988a122efe9287fb10cb04e6fe11cb"
server_key_version: "encryptionkey@0"
I20260810 18:05:53.580533 20137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/wal/instance:
uuid: "5161b8ecb2bf4557a3e0b2db6683ce47"
format_stamp: "Formatted at 2026-08-10 18:05:53 on dist-test-slave-srgv"
server_key: "030a7b59556f5cf6cf7507ac04532196"
server_key_iv: "bc988a122efe9287fb10cb04e6fe11cb"
server_key_version: "encryptionkey@0"
I20260810 18:05:53.584499 20137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 18:05:53.587214 20152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:05:53.588562 20137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:05:53.588701 20137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/data,/tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/wal
uuid: "5161b8ecb2bf4557a3e0b2db6683ce47"
format_stamp: "Formatted at 2026-08-10 18:05:53 on dist-test-slave-srgv"
server_key: "030a7b59556f5cf6cf7507ac04532196"
server_key_iv: "bc988a122efe9287fb10cb04e6fe11cb"
server_key_version: "encryptionkey@0"
I20260810 18:05:53.588835 20137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:05:53.611174 20137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:05:53.614332 20137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:05:53.614542 20137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:05:53.622298 20137 rpc_server.cc:307] RPC server started. Bound to: 127.28.83.124:43985
I20260810 18:05:53.622594 20205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.83.124:43985 every 8 connection(s)
I20260810 18:05:53.623409 20137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/data/info.pb
I20260810 18:05:53.626168 29005 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EWyHF/build/debug/bin/kudu as pid 20137
I20260810 18:05:53.626281 29005 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EWyHF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786384863806381-29005-0/minicluster-data/master-2/wal/instance
I20260810 18:05:53.626582 29005 external_mini_cluster.cc:1428] Setting key 292051737f4576dce55f2d862e790bbc
I20260810 18:05:53.626724 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:05:53.626870 20051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } attempt: 3
I20260810 18:05:53.633358 20206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.126@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
I20260810 18:05:53.640066 20128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } attempt: 3
I20260810 18:05:53.642055 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.627363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38374)
0810 18:05:53.627612 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.627629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:53.627728 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:53.630423 (+ 2695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:53.630426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:53.630663 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:53.630778 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.630786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.632098 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.632104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.632727 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.632738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.633750 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.633754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.633865 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.634670 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:53.634687 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:53.636583 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:53.639113 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.639118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.639122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.639415 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.639965 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.639971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.639975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.640091 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.641501 (+ 1410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:53.641504 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:53.641605 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 18:05:53.641708 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 18:05:53.641860 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:53.644011 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.628238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.28.83.124:43985)
0810 18:05:53.628724 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.628741 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.628777 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:05:53.629670 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.629683 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.630108 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.631463 (+ 1355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.631591 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.632502 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.633752 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.634111 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.634232 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.637227 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.637232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.637236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.637264 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.640016 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.640718 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.640723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.640725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.640813 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.641159 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.641162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.641164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.641388 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.642328 (+ 940us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.642513 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.642943 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":330,"spinlock_wait_cycles":29824,"thread_start_us":160,"threads_started":1}
I20260810 18:05:53.644011 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.628721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.28.83.124:43985)
0810 18:05:53.628989 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.628998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.629009 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:05:53.629655 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.629679 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.630096 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.630477 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.630920 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.631938 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.632870 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.633600 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.633848 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.636815 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.636820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.636833 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.636861 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.639008 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.639563 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.639569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.639575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.639775 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.640266 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.640270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.640371 (+ 101us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.641334 (+ 963us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.642350 (+ 1016us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.642513 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.642892 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":210,"spinlock_wait_cycles":19328,"thread_start_us":88,"threads_started":1}
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.125@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
I20260810 18:05:53.648981 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.635793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.28.83.126:43309)
0810 18:05:53.636233 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.636238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.636414 (+ 176us) server_negotiation.cc:407] Connection header received
0810 18:05:53.636620 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.636623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.636680 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.636731 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.637755 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.638454 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.639876 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.640261 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.640403 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.644136 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.644140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.644143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.644170 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.646402 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.647083 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.647090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.647095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.647202 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.647893 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.647898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.647900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.648143 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.648280 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.648662 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.648799 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352}
I20260810 18:05:53.649005 20051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca: Bootstrap starting.
I20260810 18:05:53.649528 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.635788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.126:43309 (local address 127.0.0.1:33966)
0810 18:05:53.636271 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.636333 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:53.636505 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:53.636913 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:53.636919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:53.637446 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:53.637607 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.637617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.638584 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.638590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.639740 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.639755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.640562 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.640566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.640724 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.641329 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:53.641356 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:53.643954 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:53.646562 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.646576 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.646586 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.646932 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.647316 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.647319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.647321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.647429 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.648279 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:53.648286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:53.648419 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 18:05:53.648599 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 18:05:53.648805 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":1,"spinlock_wait_cycles":2560,"thread_start_us":154,"threads_started":1}
I20260810 18:05:53.650831 20206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:53.652709 20051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:53.653781 20051 log.cc:824] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:53.653875 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.640627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38376)
0810 18:05:53.641310 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.641331 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:53.641469 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:53.643154 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:53.643159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:53.643377 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:53.643516 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.643529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.645438 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.645446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.646470 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.646486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.647203 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.647208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.647319 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.647820 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:53.647836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:53.649323 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:53.651975 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.651978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.651980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.652249 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.652606 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.652609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.652613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.652711 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.653255 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:53.653260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:53.653377 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 18:05:53.653553 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 18:05:53.653708 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260810 18:05:53.654793 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.640918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.28.83.124:43985)
0810 18:05:53.642849 (+ 1931us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.642863 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.642882 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:05:53.642924 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.642927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.642976 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.644135 (+ 1159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.644278 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.645254 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.646641 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.647059 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.647581 (+ 522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.649529 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.649533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.649535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.649560 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.651847 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.652376 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.652381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.652385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.652488 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.652843 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.652846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.652848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.653074 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.653184 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.654479 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.654616 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
I20260810 18:05:53.658402 20051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca: No bootstrap required, opened a new log
I20260810 18:05:53.659090 20128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039: Bootstrap starting.
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.126@KRBTEST.COM
I20260810 18:05:53.660753 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.651471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.28.83.125:43307)
0810 18:05:53.651617 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.651624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.651657 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:05:53.651772 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.651775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.651830 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.651915 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.652342 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.653220 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.654341 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.654579 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.654713 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.656931 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.656935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.656937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.656962 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.658668 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.659236 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.659240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.659242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.659330 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.659653 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.659656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.659658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.659935 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.660022 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.660431 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.660565 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:05:53.661350 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.651400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.125:43307 (local address 127.0.0.1:37788)
0810 18:05:53.651592 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.651607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:53.651685 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:53.651929 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:53.651932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:53.652132 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:53.652215 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.652220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.653347 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.653350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.654213 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.654224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.654716 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.654720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.654811 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.655295 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:53.655306 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:53.656811 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:53.658811 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.658814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.658816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.659096 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.659459 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.659462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.659464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.659548 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.660107 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:53.660110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:53.660234 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 18:05:53.660989 (+ 755us) client_negotiation.cc:240] Negotiation successful
0810 18:05:53.661147 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260810 18:05:53.662196 20206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:05:53.662526 20128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039: Neither blocks nor log segments found. Creating new log.
I20260810 18:05:53.663334 20051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } }
I20260810 18:05:53.663473 20128 log.cc:824] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:53.663584 20051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8263608987fd4c3080904413ceb76bca, State: Initialized, Role: FOLLOWER
I20260810 18:05:53.663937 20051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } }
I20260810 18:05:53.664604 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.646863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.28.83.126:43309)
0810 18:05:53.647149 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.647175 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.653502 (+ 6327us) server_negotiation.cc:407] Connection header received
0810 18:05:53.653552 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.653555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.653607 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.654619 (+ 1012us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.654739 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.655539 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.656746 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.657121 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.657244 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.660353 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.660360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.660366 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.660390 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.662376 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.663111 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.663115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.663117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.663219 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.663522 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.663525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.663527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.663762 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.663975 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.664244 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.664379 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":72,"threads_started":1}
I20260810 18:05:53.665710 20051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:53.667451 20128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039: No bootstrap required, opened a new log
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.83.124@KRBTEST.COM for kudu/127.28.83.124@KRBTEST.COM
Aug 10 18:05:53 dist-test-slave-srgv krb5kdc[19961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.83.125@KRBTEST.COM
I20260810 18:05:53.672176 20128 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } }
I20260810 18:05:53.672417 20128 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b42a1c198544c9949527df17a20039, State: Initialized, Role: FOLLOWER
I20260810 18:05:53.672797 20128 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } }
I20260810 18:05:53.673857 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.662755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.83.124:43985 (local address 127.0.0.1:38386)
0810 18:05:53.662866 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 18:05:53.662882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:05:53.662961 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:05:53.664345 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:05:53.664347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:05:53.664514 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:05:53.664598 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.664604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.665583 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.665586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.666324 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:05:53.666335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.667125 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.667129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:05:53.667220 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.667922 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:05:53.667937 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:05:53.669771 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:05:53.672013 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.672016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.672018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.672313 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.672582 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:05:53.672584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:05:53.672586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:05:53.672657 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:05:53.673056 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:05:53.673059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:05:53.673132 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 18:05:53.673239 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 18:05:53.673679 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 18:05:53.674468 20128 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:53.674646 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.662824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.28.83.124:43985)
0810 18:05:53.663985 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.663999 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.664014 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:05:53.664073 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.664075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.664122 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.664219 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.664716 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.665471 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.666460 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.666832 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.667628 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.669907 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.669910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.669912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.669934 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.671889 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.672419 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.672422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.672424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.672496 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.672741 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.672745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.672748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.672940 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.673025 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.674369 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.674505 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077}
I20260810 18:05:53.671072 19975 master_runner.cc:186] Error getting master registration for 127.28.83.125:43307: OK, instance_id { permanent_uuid: "c1b42a1c198544c9949527df17a20039" instance_seqno: 1786385153386287 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:05:53.675571 19975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:05:53.668107 20216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.675753 20216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:53.677588 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.665975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.28.83.125:43307)
0810 18:05:53.666082 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.666086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.666293 (+ 207us) server_negotiation.cc:407] Connection header received
0810 18:05:53.666461 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.666463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.666506 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.666554 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.667683 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.668379 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.669579 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.669910 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.669989 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.673109 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.673116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.673122 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.673153 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.674955 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.676002 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.676006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.676007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.676098 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.676442 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.676448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.676450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.676719 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.676897 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.677289 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.677444 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:05:53.677919 20206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47: Bootstrap starting.
I20260810 18:05:53.677953 20230 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.678097 20230 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:53.680831 20206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47: Neither blocks nor log segments found. Creating new log.
W20260810 18:05:53.681092 20241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:53.681190 20241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 18:05:53.681444 20229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:53.681597 20229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:05:53.681603 29005 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:05:53.682567 20206 log.cc:824] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47: Log is configured to *not* fsync() on all Append() calls
I20260810 18:05:53.682971 20023 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8263608987fd4c3080904413ceb76bca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33974
I20260810 18:05:53.683257 20023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:05:53.683420 20023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:53.684543 20023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } }
I20260810 18:05:53.685423 20023 leader_election.cc:288] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [CANDIDATE]: Term 1 election: Requested vote from peers c1b42a1c198544c9949527df17a20039 (127.28.83.125:43307), 5161b8ecb2bf4557a3e0b2db6683ce47 (127.28.83.124:43985)
I20260810 18:05:53.686030 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8263608987fd4c3080904413ceb76bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c1b42a1c198544c9949527df17a20039"
I20260810 18:05:53.686298 20101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:53.688340 20101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8263608987fd4c3080904413ceb76bca in term 1.
I20260810 18:05:53.689931 19996 leader_election.cc:302] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8263608987fd4c3080904413ceb76bca, c1b42a1c198544c9949527df17a20039; no voters:
I20260810 18:05:53.690264 20216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:05:53.690511 20216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [term 1 LEADER]: Becoming Leader. State: Replica: 8263608987fd4c3080904413ceb76bca, State: Running, Role: LEADER
I20260810 18:05:53.690929 20216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } }
I20260810 18:05:53.692857 20216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8263608987fd4c3080904413ceb76bca. Latest consensus state: current_term: 1 leader_uuid: "8263608987fd4c3080904413ceb76bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.692968 20216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:53.694229 20254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:05:53.691781 20058 master_runner.cc:186] Error getting master registration for 127.28.83.124:43985: OK, instance_id { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" instance_seqno: 1786385153620048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:05:53.695413 20058 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:05:53.698971 20254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:05:53.700832 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.689672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.28.83.125:43307)
0810 18:05:53.690319 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.690323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.690338 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:05:53.690373 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.690376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.690418 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.690497 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.691907 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.692642 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.693832 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.694153 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.694245 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.696267 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.696270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.696273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.696301 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.698032 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.698898 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.698902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.698904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.698996 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.699502 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.699506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.699508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.699748 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.699870 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.700526 (+ 656us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.700659 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568}
I20260810 18:05:53.700863 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.688726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.28.83.124:43985)
0810 18:05:53.688882 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.688886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.689609 (+ 723us) server_negotiation.cc:407] Connection header received
0810 18:05:53.689852 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.689857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.689910 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.689988 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.691157 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.691917 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.693563 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.693903 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.694016 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.697044 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.697060 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.697063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.697095 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.698851 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.699419 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.699422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.699424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.699521 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.699853 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.699856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.699858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.700076 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.700214 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.700568 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.700709 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:05:53.705358 20206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47: No bootstrap required, opened a new log
I20260810 18:05:53.711257 20206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } }
I20260810 18:05:53.711571 20206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5161b8ecb2bf4557a3e0b2db6683ce47, State: Initialized, Role: FOLLOWER
I20260810 18:05:53.711709 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:53.688329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33978 (local address 127.28.83.126:43309)
0810 18:05:53.688460 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.688465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.701575 (+ 13110us) server_negotiation.cc:407] Connection header received
0810 18:05:53.701819 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.701823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.701897 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.701975 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.703252 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.704005 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.705104 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:53.705502 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:53.705597 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:53.707564 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:05:53.707569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:05:53.707572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:05:53.707602 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:05:53.709468 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.710104 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.710109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.710112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.710220 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:05:53.710568 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:05:53.710571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:05:53.710573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:05:53.710803 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:05:53.710907 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:53.711371 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 18:05:53.711512 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:05:53.711936 20206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } }
I20260810 18:05:53.713858 20206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:05:53.713807 20256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.713996 20256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:53.713924 20180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8263608987fd4c3080904413ceb76bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47"
I20260810 18:05:53.714206 20180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:05:53.715693 20180 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8263608987fd4c3080904413ceb76bca in term 1.
I20260810 18:05:53.718706 20101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [term 1 FOLLOWER]: Refusing update from remote peer 8263608987fd4c3080904413ceb76bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:53.719440 20252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:53.719619 20180 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [term 1 FOLLOWER]: Refusing update from remote peer 8263608987fd4c3080904413ceb76bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:05:53.720175 20252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:05:53.725005 20256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8263608987fd4c3080904413ceb76bca. Latest consensus state: current_term: 1 leader_uuid: "8263608987fd4c3080904413ceb76bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.725147 20256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:53.725562 20230 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8263608987fd4c3080904413ceb76bca. Latest consensus state: current_term: 1 leader_uuid: "8263608987fd4c3080904413ceb76bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.725695 20230 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 18:05:53.729029 20267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:05:53.729187 20267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:05:53.734247 20252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8263608987fd4c3080904413ceb76bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.734421 20252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:53.736582 20230 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8263608987fd4c3080904413ceb76bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.736726 20230 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:53.738777 20268 mvcc.cc:205] Tried to move back new op lower bound from 7317033589617676288 to 7317033589521702912. Current Snapshot: MvccSnapshot[applied={T|T < 7317033589617676288 or (T in {7317033589617676288})}]
I20260810 18:05:53.746508 20254 catalog_manager.cc:1347] Generated new cluster ID: be3d124138eb4309a6e3bc25e490679c
I20260810 18:05:53.746614 20254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:05:53.748169 20216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8263608987fd4c3080904413ceb76bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.748358 20216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca [sys.catalog]: This master's current role is: LEADER
I20260810 18:05:53.750913 20269 mvcc.cc:205] Tried to move back new op lower bound from 7317033589617676288 to 7317033589521702912. Current Snapshot: MvccSnapshot[applied={T|T < 7317033589617676288}]
I20260810 18:05:53.757531 20256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8263608987fd4c3080904413ceb76bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8263608987fd4c3080904413ceb76bca" member_type: VOTER last_known_addr { host: "127.28.83.126" port: 43309 } } peers { permanent_uuid: "c1b42a1c198544c9949527df17a20039" member_type: VOTER last_known_addr { host: "127.28.83.125" port: 43307 } } peers { permanent_uuid: "5161b8ecb2bf4557a3e0b2db6683ce47" member_type: VOTER last_known_addr { host: "127.28.83.124" port: 43985 } } }
I20260810 18:05:53.757723 20256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:05:53.763721 20254 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:05:53.765784 20254 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:05:53.783339 20254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8263608987fd4c3080904413ceb76bca: Generated new TSK 0
I20260810 18:05:53.784353 20254 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:05:53.820354 20213 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.816851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.28.83.126:43309)
0810 18:05:53.816988 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.816993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.818956 (+ 1963us) server_negotiation.cc:407] Connection header received
0810 18:05:53.819386 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.819390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.819449 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.819508 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.820204 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33988: BlockingRecv error: recv got EOF from 127.0.0.1:33988 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 18:05:53.821573 20212 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.817675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.28.83.124:43985)
0810 18:05:53.817819 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.817824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.819298 (+ 1474us) server_negotiation.cc:407] Connection header received
0810 18:05:53.820750 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.820754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.820806 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.820859 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.821440 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38404: BlockingRecv error: recv got EOF from 127.0.0.1:38404 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 18:05:53.823143 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.818325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.28.83.125:43307)
0810 18:05:53.818442 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.818447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.822171 (+ 3724us) server_negotiation.cc:407] Connection header received
0810 18:05:53.822399 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.822402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.822453 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.822505 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.823014 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37806: BlockingRecv error: recv got EOF from 127.0.0.1:37806 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 18:05:53.827996 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.826385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.28.83.125:43307)
0810 18:05:53.826495 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.826499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.826999 (+ 500us) server_negotiation.cc:407] Connection header received
0810 18:05:53.827224 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.827227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.827306 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.827360 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.827891 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37810: BlockingRecv error: recv got EOF from 127.0.0.1:37810 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 18:05:53.833197 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.831651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.28.83.125:43307)
0810 18:05:53.831768 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.831772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.832283 (+ 511us) server_negotiation.cc:407] Connection header received
0810 18:05:53.832469 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.832472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.832522 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.832573 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.833089 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37822: BlockingRecv error: recv got EOF from 127.0.0.1:37822 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 18:05:53.840569 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.838800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.28.83.125:43307)
0810 18:05:53.838941 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.838945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.839565 (+ 620us) server_negotiation.cc:407] Connection header received
0810 18:05:53.839790 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.839794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.839848 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.839900 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.840453 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37834: BlockingRecv error: recv got EOF from 127.0.0.1:37834 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 18:05:53.848973 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.847153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.28.83.125:43307)
0810 18:05:53.847307 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.847312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.847919 (+ 607us) server_negotiation.cc:407] Connection header received
0810 18:05:53.848132 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.848136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.848193 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.848250 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.848856 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37850: BlockingRecv error: recv got EOF from 127.0.0.1:37850 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 18:05:53.861954 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.860067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.28.83.125:43307)
0810 18:05:53.860202 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.860207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.860835 (+ 628us) server_negotiation.cc:407] Connection header received
0810 18:05:53.861058 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.861063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.861125 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.861178 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.861834 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37864: BlockingRecv error: recv got EOF from 127.0.0.1:37864 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 18:05:53.882939 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.880953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.28.83.125:43307)
0810 18:05:53.881105 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.881110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.881838 (+ 728us) server_negotiation.cc:407] Connection header received
0810 18:05:53.882070 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.882074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.882133 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.882189 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.882815 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37874: BlockingRecv error: recv got EOF from 127.0.0.1:37874 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 18:05:53.919217 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.917852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.28.83.125:43307)
0810 18:05:53.917994 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.917999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.918258 (+ 259us) server_negotiation.cc:407] Connection header received
0810 18:05:53.918428 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.918431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.918491 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.918585 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.919074 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37880: BlockingRecv error: recv got EOF from 127.0.0.1:37880 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 18:05:53.986781 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:53.985615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.28.83.125:43307)
0810 18:05:53.985773 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:53.985777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:53.985933 (+ 156us) server_negotiation.cc:407] Connection header received
0810 18:05:53.986054 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:53.986056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:53.986106 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:53.986201 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:53.986677 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37892: BlockingRecv error: recv got EOF from 127.0.0.1:37892 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 18:05:54.118283 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:54.117120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.28.83.125:43307)
0810 18:05:54.117266 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:54.117270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:54.117394 (+ 124us) server_negotiation.cc:407] Connection header received
0810 18:05:54.117589 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:54.117591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:54.117644 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:54.117735 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:54.118166 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37900: BlockingRecv error: recv got EOF from 127.0.0.1:37900 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 18:05:54.378106 20133 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:05:54.376834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.28.83.125:43307)
0810 18:05:54.376959 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:54.376963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:54.377004 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:05:54.377157 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:54.377160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:54.377208 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:54.377293 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:05:54.377977 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37904: BlockingRecv error: recv got EOF from 127.0.0.1:37904 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 18:05:54.682649 20241 catalog_manager.cc:1259] Loaded cluster ID: be3d124138eb4309a6e3bc25e490679c
I20260810 18:05:54.682766 20241 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039: loading cluster ID for follower catalog manager: success
I20260810 18:05:54.684455 20241 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039: acquiring CA information for follower catalog manager: success
I20260810 18:05:54.685672 20241 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c1b42a1c198544c9949527df17a20039: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:05:54.730672 20267 catalog_manager.cc:1259] Loaded cluster ID: be3d124138eb4309a6e3bc25e490679c
I20260810 18:05:54.730765 20267 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47: loading cluster ID for follower catalog manager: success
I20260810 18:05:54.732545 20267 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47: acquiring CA information for follower catalog manager: success
I20260810 18:05:54.733790 20267 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5161b8ecb2bf4557a3e0b2db6683ce47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:05:54.896304 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:54.892332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.28.83.125:43307)
0810 18:05:54.892583 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:54.892586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:54.892612 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:05:54.892746 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:54.892748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:54.892891 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:54.893052 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:54.893329 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.893894 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.894748 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.894975 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.895058 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:54.895355 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:54.895806 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:54.895904 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:54.896092 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 18:05:54.896172 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
I20260810 18:05:54.903427 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:54.899660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.28.83.124:43985)
0810 18:05:54.899922 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:54.899925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:54.899962 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:05:54.900044 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:54.900047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:54.900171 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:54.900267 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:54.900514 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.901048 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.901901 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.902136 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.902226 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:54.902540 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:54.902974 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:54.903068 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:54.903167 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 18:05:54.903219 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260810 18:05:54.911127 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:54.906603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.28.83.126:43309)
0810 18:05:54.906863 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:54.906869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:54.906886 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:05:54.907070 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:54.907076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:54.907254 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:54.907396 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:54.907614 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.908347 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.909515 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.909879 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.909988 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:54.910327 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:54.910705 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:54.910816 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:54.910897 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 18:05:54.910956 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
I20260810 18:05:54.911162 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:54.906711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.28.83.125:43307)
0810 18:05:54.906865 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:54.906869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:54.907249 (+ 380us) server_negotiation.cc:407] Connection header received
0810 18:05:54.907443 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:54.907445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:54.907500 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:54.907553 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:54.907932 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.908750 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.909894 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.910256 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.910382 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:54.910644 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:54.910727 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:54.910795 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:54.910959 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 18:05:54.911033 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:05:54.911212 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:05:54.907155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.28.83.124:43985)
0810 18:05:54.907289 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:05:54.907294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:05:54.907460 (+ 166us) server_negotiation.cc:407] Connection header received
0810 18:05:54.907658 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:05:54.907661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:05:54.907713 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:05:54.907786 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:05:54.908053 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.908804 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.909968 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:05:54.910279 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:05:54.910354 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:05:54.910732 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:05:54.910827 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:05:54.910895 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:05:54.911037 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 18:05:54.911088 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:05:54.914784 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 19975
I20260810 18:05:54.922542 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 20058
I20260810 18:05:54.929342 29005 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EWyHF/build/debug/bin/kudu with pid 20137
2026-08-10T18:05:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3148 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6323 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (291080 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST